A Failure of Aspect Sentiment Classifiers and an Adaptive Re-weighting Solution
Table 4: Performance of ASC baselines and the proposed ARW Scheme on both Full Test Set and Contrastive Test Set; the BERT-DK model is further tested on examples by removing aspects as in (on Full Test Set w/o aspect).
['[EMPTY]', '[BOLD] Laptop Acc.', 'MF1', '[BOLD] Rest. Acc.', 'MF1']
, we can see those existing ASC classifiers perform poorly on the contrastive test sets, which contain real ASC examples only. To answer RQ1, we find that all baselines have significant drops on both Accuracy (Acc.) and F1 score as most existing models reach more than 70% on both accuracy and F1 on the full test set. Lastly, when the aspects are dropped from the input (on Full Test Set w/o aspect), the BERT-DK ASC classifier dropped a little and still comparable to other baselines on the full test set. Since this experiment has no access to aspects but just the review sentences, it indicates that the model DOES NOT count on aspects much in doing aspect-level sentiment classification. To answer RQ2, we observe that manual re-weighting improves the performance on laptop and restaurant about 3% for the contrastive test sets. After manual re-weighting, the performance on the full test set improves on restaurant but drops on laptop slightly. The reason could be that manual weights are not perfect for learning, which may overemphasize rare examples from contrastive sentences in the laptop training data.