MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension
Table 10: Error analysis on DREAM. The column of “Percent” reports the percentage of question types among 150 samples that are from the development set of DREAM dataset that are wrongly predicted by the BERT-Base baseline model. The column of “Accuracy” reports the accuracy of our best model (RoBERTa-Large+MMM) on these samples.
['[BOLD] Major Types', '[BOLD] Sub-types', '[BOLD] Percent', '[BOLD] Accuracy']
In order to investigate how well our model performs for different types of questions, we did an error analysis by first randomly selecting 150 samples from the common wrong predictions on the development set of DREAM dataset, obtained by three BERT-Base baseline models, each of which was individually trained with different random seeds. We see that the BERT-Base baseline model still does not do well on matching problems. We find that our best model can improve upon every question type significantly especially for the matching problems, and most surprisingly, our best model can even greatly improve its ability on solving the arithmetic problems, achieving the accuracy of 73.7%.