Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
Table 5: Accuracy (%) of different models on the WSC dataset together with its subsets and the WNLI test set. MTP denotes masked token prediction, which is employed in Kocijan et al. (2019). MCQA denotes the multi-choice question-answering format, which is employed in this paper.
['[BOLD] Model', '[BOLD] WSC', '[BOLD] non-assoc.', '[BOLD] assoc.', '[BOLD] unswitched', '[BOLD] switched', '[BOLD] consist.', '[BOLD] WNLI']
Note that non-assoc. denotes instances in WSC that no antecedent is statistically preferred, and assoc. denotes the rest instances. denotes the percentage of predictions that change after candidates in the switchable subset are switched. Higher scores for non-assoc. Note that the results for Ruan et al. Results for “BERTlarge + MTP” is cited from Kocijan et al. As can be seen from the table, our “BERT_CSlarge + MCQA” achieves the best performance on all of the evaluation criteria except being the second best on consist. Further analysis shows that “BERTlarge + MCQA” achieves better performance than “BERTlarge + MTP” on the WSC and WNLI testsets and achieves significant improvements on consist., suggesting that MCQA may be a better problem formatting method than MTP for the WSC task. However, these WSC results demonstrate that the proposed approach is general for improving commonsense reasoning capabilities of the pre-trained models.