Scalable Syntax-Aware Language Models Using Knowledge Distillation
Table 1: Replication of marvin_2018 results. M&L-Multi is the marvin_2018 LSTM trained on LM and CCG supertagging Bangalore and Joshi (1999); Clark and Curran (2007) losses with an interpolation factor of 0.5. We report our LSTM LM, small LSTM†, and RNNG† performance (†smaller training data; §3) in the format of mean (±standard deviation) of 10 identical models from different seeds. Results in bold denote the best among models trained on similar amounts of training data.
['[EMPTY]', '[BOLD] Marvin & Linzen models  [BOLD] M&L-LSTM', '[BOLD] Marvin & Linzen models  [BOLD] M&L-Multi', '[BOLD] Ours  [BOLD] Our LSTM', '[BOLD] Ours (small training)  [BOLD] Small LSTM†', '[BOLD] Ours (small training)  [BOLD] RNNG†', '[BOLD] Humans']
Our LSTM LM achieves much better perplexity than the LSTM LM (32% ppl. reduction) and even the multi-task LSTM (12% reduction) of \newcitemarvin_2018. As our LSTM has the same number of hidden units, we attribute this gap to differences in optimisation and codebases. On aggregate, our LSTM LM outperforms the LSTM multi-task model from \newcitemarvin_2018 that exploits explicit CCG annotations, and is able to match or exceed human performance on 7 out of all 15 constructions, thus confirming earlier findings that neural language models are able to acquire complex syntactic generalisation without explicit syntactic supervision Gulordava et al.