Detecting Local Insights from Global Labels: Supervised & Zero-Shot Sequence Labeling via a Convolutional Decomposition
Table 12: Results for predicting sentiment on out-of-domain data, the SemEval-2017 Task 4a test set. We have balanced the set and dropped the third class (neutral) for consistency with earlier work evaluating with accuracy, resulting in 4,750 Twitter messages for evaluation. Results are with the uniCNN+BERT model, except for the rows marked BASE_uncased, which are from the uniCNN+BERTBASE_uncased model (included here to check whether the sign flips relative to previous work are a pathology of the cased model on the Twitter data). We vary the training data with regard to the amount of original data and whether or not revised reviews are included. (Twitter data is not seen in training, nor for development/tuning.) Across models and training set sizes, adding the revised data (with parallel source and target) does not consistently improve effectiveness on this out-of-domain data. The implications of the disjoint (non-parallel source and target) data results are discussed further in the text.
['[EMPTY]', 'Review-level Sentiment (Accuracy)']
Interestingly, with our uniCNN+BERT model, which uses a cased BERT_LARGE model, accuracy drops significantly when using parallel source-target pairs, relative to training with disjoint pairs, but also compared to just training with Orig. data. This is the opposite of what was observed in previous work. Following the results above (in particular, the observation that other things being equal, training with parallel pairs leads to higher precision, at least with the standard decision boundary), we speculate that the model learns to especially emphasize the tokens that only participate in transduction deletions, insertions, and replacements. Since the model is cased, relatively few of these features will appear verbatim in the Twitter data.