Scalable Syntax-Aware Language Models Using Knowledge Distillation
Table 2: Experimental findings of the “DSA-LSTM”. For each column, we report the mean of 10 identical models trained from different random seeds; standard deviation values are reported in the Appendix. “S-DSA-LSTM” indicates the DSA-LSTM trained on the smaller RNNG training set, while “BA-LSTM” is the born-again model where the teacher is the full LSTM LM. We also compare with the syntactic generalisation of “BERT” Base Devlin et al. (2019); Goldberg (2019), which is not strictly comparable since it is trained on 30 times as much data. † indicates models trained on the smaller 20% training set (§3). Results in bold denote the best among those trained with the same amounts of data.
['[EMPTY]', '[BOLD] Small Training Set  [BOLD] Small LSTM†', '[BOLD] Small Training Set  [BOLD] S-DSA-LSTM†', '[BOLD] Small Training Set  [BOLD] RNNG†', '[BOLD] Full Training Set  [BOLD] Full LSTM', '[BOLD] Full Training Set  [BOLD] BA-LSTM', '[BOLD] Full Training Set  [BOLD] DSA-LSTM', '[BOLD] BERT', '[BOLD] Humans']
Despite its smaller training set, the RNNG performs extremely well on subject-verb agreement, substantially outperforming both the full LSTM and a pre-trained BERT (Devlin et al.,