Detecting Local Insights from Global Labels: Supervised & Zero-Shot Sequence Labeling via a Convolutional Decomposition
Table 13: Accuracy results for predicting original (Orig.) vs. revised (Rev.) data on the test set using the uniCNN+BERT model. The task here is to predict the domain (i.e., whether or not a review is a “counterfactual” re-write), and not to predict sentiment. We include additional results subdivided by sentiment and the domain classes. Any mistake in the Orig. subset is a false positive, and any mistake in the Rev. subset is a false negative. (The accuracy of a random predictor is around 50 for each subset.) The revised reviews and the negative reviews tend to be easier to identify as being revised, as shown by the higher accuracies for those subsets.
['Test (Sub-)Set', 'Review-level Domain ( [ITALIC] Not Sentiment) Accuracy', 'Review-level Domain ( [ITALIC] Not Sentiment) Num. Reviews']
We find that the original reviews are relatively easy to distinguish from counterfactually revised reviews with the uniCNN+BERT model. On the 976 reviews of the combined Orig.+Rev. test set, about 80% of reviews are correctly distinguished as being original vs. revised reviews. The revised reviews are slightly easier to distinguish in general (accuracy of 80.5 vs. 78.7), but the main divergence comes when conditioning on sentiment, with the negative reviews noticeably easier to distinguish as being revised. For example, in the combined set, the accuracy is nearly 9 points higher on the negative reviews (accuracy of 84.0 vs. 75.2). This suggests that the language used in the negative reviews is particularly distinctive between the original and revised reviews, which we explore further with the text analysis methods proposed here.