A Failure of Aspect Sentiment Classifiers and an Adaptive Re-weighting Solution
Table 3: Summary of Contrastive Test Set.
['[EMPTY]', '[BOLD] Laptop', '[BOLD] Restaurant']
What is worse is that the test set for the laptop domain contains only 38 contrastive sentences. This further poses a challenge on evaluating the ASC capability for laptop as only contrastive sentences can evaluate the true capability of ASC. To solve this problem, two (2) annotators are asked to follow the annotation instructions of SemEval 2014 Task 4 and annotate more contrastive sentences (to have a similar number of contrastive sentences as restaurant in total) from Laptop reviews Disagreements are discussed until agreements are reached. The main complaint from the annotators is that finding such sentences takes a lot of time as they are infrequent. By combining the additional contrastive sentences with those contrastive sentences from the original test set, we form a new contrastive test set, dedicated to testing the true ASC capability of ASC classifiers. Note that there is no change to the training set for the laptop domain and no change to either the training or the test set of the restaurant domain. To simplify our description, we refer the original test set as the full test set. Note that we DO NOT add those extra contrastive sentences into the full test set to keep the results comparable with existing approaches. We evaluate the failure of existing ASC classifiers on the contrastive test set in experiments and discuss our example re-weighting scheme that focuses training on rare contrastive sentences in the next section.