Unsupervised Recurrent Neural Network Grammars
Table 2: (Top) Comparison of this work as a language model against prior works on sentence-level PTB with preprocessing from Dyer et al. (2016). Note that previous versions of RNNG differ from ours in terms of parameterization and model size. (Bottom) Results on a subset (1M sentences) of the one billion word corpus. PRPN† is the model from Shen et al. (2018), whose hyperparameters were tuned by us. RNNG‡ is trained on predicted parse trees from the self-attentive parser from Kitaev and Klein (2018).
['PTB', 'PPL']
We find that a standard language model (RNNLM) is better at modeling short sentences, but underperforms models that explicitly take into account structure (RNNG/URNNG) when the sentence length is greater than 10. On this larger dataset URNNG still improves upon the RNNLM. We also trained an RNNG (and RNNG → URNNG) on this dataset by parsing the training set with the self-attentive parser from Kitaev and Klein These models improve upon the RNNLM but not the URNNG, potentially highlighting the limitations of using predicted trees for supervising RNNGs.