Attention-Guided Answer Distillation for Machine Reading Comprehension
Table 5: Ablation study of different knowledge distillation approaches on SQuAD dev set and AddSent set.
['[EMPTY]', 'Dev F1', 'Dev ΔF1', 'AddSent F1', 'AddSent ΔF1']
First, we evaluate the vanilla knowledge distillation by removing the KD loss. We find that this loss contributes a lot on both datasets, implying that standard knowledge distillation is helpful for MRC models. Next we ablate the attention loss (ATT), and observe a smaller performance degradation on both datasets, which suggests that matching attention distributions is also beneficial but less effective. We then test the effect of removing answer distillation (ANS), and discover that although the impact on the SQuAD dev set is relatively small, the influence on the AddSent dataset is quite large. We argue that this is because the model trained with answer distillation can better handle the biased distillation problem, which is more severe in the adversarial dataset. Finally, we replace the joint training process with a stage-wise fashion proposed by \newciteRomero15. Concretely, we first warm up the student by matching the attention distribution as a pre-training step, and then minimize the rest of losses to train the model. The result, however, shows that this strategy does harm to the performance. We think the reason may be that the pre-training step leads the model into a local minima.