Story Ending Prediction by Transferable BERT
Table 3: Experimental results of story ending prediction on SCT_v1.0 test dataset. Differences between our best method and all baseline methods are significant (p<0.01) using t-test. BERTLARGE + MNLI also gets the SOTA performance of 90.3% on the newly released SCT_v1.5 blind test dataset, which is not shown in this table.
['[BOLD] Method', '[BOLD] Accuracy (%)']
The best previously reported result is from ISCK This is because the BERT model can obtain general language knowledge from pre-training. The MNLI-enhanced BERT models achieved the best accuracies of 90.5% and 91.8%, which are the new state-of-the-art performances on SCT_v1.0. This is because our method can learn task-specific knowledge from transfer tasks, which is helpful for SCT, and MNLI is the most informative task. Comparing the SNLI and MNLI-enhanced BERT models, we find that though NLI can help SCT intuitively, the data source still plays an important role. MNLI is a multi-genre dataset, while SNLI data is from the specific image caption domain. Hence, the MNLI tends to help the open domain SCT but SNLI does not. Comparing the IMDB and Twitter-enhanced BERT models, we can get similar conclusions that the open domain Twitter can improve the performance of BERT on SCT, while the specific domain IMDB hurts the model’s performance. Comparing the MC_MNLI and MNLI-enhanced BERT models, we find that MNLI helps more for SCT (multiple choice task). Hence, we can get the conclusion that the transfer task doesn’t need to have the same problem definition as the target task. This is mainly because the model can get a better knowledge about entailment when NLI is formulated as a classification task (MNLI), other than a multiple choice task (MC_MNLI).