Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
Table 4: Accuracy (%) of different models on the CSQA test set.
['[BOLD] Model', '[BOLD] Accuracy']
Same as the pre-training stage, the input data for fine-tuning BERT_CS is formed by concatenating each question-answer pair as a sequence. The hidden representations over the [CLS] token are run through a softmax layer to create the predictions. We fine-tune BERT_CS on CSQA for 2 epochs with a learning rate of 1e-5 and a batch size of 16. CoS-E model requires a large amount of human efforts to collect the Common Sense Explanations (CoS-E) dataset. In comparison, our multi-choice QA dataset DAMS is constructed automatically. The BERT_CS models significantly outperform the baseline BERT models with BERT_CSlarge achieving 5.5% absolute gain over the baseline BERTlarge model and 4.0% absolute gain over the previous SOTA CoS-E model.