Attention-Guided Answer Distillation for Machine Reading Comprehension
Table 1: Comparison of different approaches on the SQuAD test set, extracted on May 9, 2018: Rajpurkar161, Huang17b2, Elmo183, Wang17b4, wang2018multi5, Yu186 and Hu177. BiSAE refers to BiDAF + Self Attention + ELMo. (E: ensemble model)
['Model', 'Dev EM', 'Dev F1', 'Test EM', 'Test F1']
We can see that the student network is able to compete with the ensemble model with only a slight drop of 0.4% on F1. Moreover, nearly 80% of the improvement in terms of F1 achieved by the ensemble model is successfully transferred to the distilled model, which indicates the effectiveness of our approach.