Detecting Local Insights from Global Labels: Supervised & Zero-Shot Sequence Labeling via a Convolutional Decomposition
Table 6: Accuracy results for predicting sentiment on the original (Orig.) and revised (Rev.) test sets. These are reference results placing the proposed models in the context of previous work. These models are all trained on the full original training set (19k) and the revised training set (1.7k). The results for BERTBASE_uncased+FT, which fine-tunes the BERTBASE_uncased parameters, are those of Kaushik, Hovy, and Lipton (2019). Interestingly, using the frozen weights of BERTBASE_uncased or BERTBASE with the BLADE layer (and non-static word embeddings) only modestly decreases accuracy, despite not updating the 110 million BERTBASE parameters, saving considerable computational resources for training. The uniCNN+BERT model is comparable to BERTBASE_uncased+FT, and we use this uniCNN+BERT model for our analyses of the datasets.
['Model', 'Review-level Sentiment (Accuracy) Orig.', 'Review-level Sentiment (Accuracy) Rev.']
4.2.1 Predicting Sentiment (Negative vs. Positive) Interestingly, the fall in effectiveness is only 1 to 3 points, without updating the 110 million parameters. As noted above, we assume that this gap could be closed by fine-tuning the BERT parameters with an imputation loss and then training the BLADE layer as before; however, to minimize computation, and since we are primarily concerned with internal validity for our analyses of the data, we instead leverage this result that using the frozen BERT parameters with the BLADE layer still yields strong results. This is a critical practical observation, because without the need to update the gradients of the BERT model, we can readily move to the BERT_LARGE model, allowing us to run the many experiments of this work with the uniCNN+BERT model (which yields comparable results to fine-tuning the BERTBASE model) using a relatively modest amount of computation for training. To investigate this further, we also evaluate with the uniCNN+BERTBASE_uncased model. With the uncased model, the drop is less severe, but it is only with the larger dataset (Orig.+Rev. (19k+1.7k) BASE_uncased) that the accuracy is improved over just training with the original dataset. We speculate that the addition of the non-parallel original sentences in that training split moderates the emphasis on tokens participating in diffs. We lack token-level Twitter labels to verify this behavior, but it seems reasonable given the results above with the sentiment diffs. With the disjoint pairs and the uncased models, the results are similar to training on the original data with the 3.4k set and exceed the corresponding original result by several points on the larger set, suggesting that training with a disjoint set (e.g., from another domain) may be helpful under significant domain shift (which is consistent with what we might otherwise assume).