TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing
Table 5: Development set results for the teacher and various students on Chinese tasks.
['[BOLD] Model', '[BOLD] XNLI Acc', '[BOLD] LCQMC Acc', '[BOLD] CMRC 2018 EM', '[BOLD] CMRC 2018 F1', '[BOLD] DRCD EM', '[BOLD] DRCD F1']
We notice that T4-tiny still outperforms T3-small on all tasks, which is consistent with their performance on English tasks. In the experiments with DA, CMRC 2018 and DRCD take each other’s dataset as data augmentation. We observe that since CMRC 2018 has a relatively small training set, DA has a much more significant effect.