TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing
Table 2: Performance of BERT\tt BASE (teacher) and various students on the development sets of MNLI and SQuAD, and the test set of CoNLL-2003. m and mm under MNLI denote the accuracies on matched and mis-matched sections respectively.
['[BOLD] Model', '[BOLD] MNLI m', '[BOLD] MNLI mm', '[BOLD] SQuAD EM', '[BOLD] SQuAD F1', '[BOLD] CoNLL-2003 F1']
We have the following observations. Finally, data augmentation (DA) is critical. training set is used in SQuAD; passages from the HotpotQA Yang et al. training set is used in CoNLL-2003. The augmentation datasets significantly improve the performance, especially when the size of the training set is small, like CoNLL-2003.