Unsupervised Recurrent Neural Network Grammars
Table 1: Language modeling perplexity (PPL) and grammar induction F1 scores on English (PTB) and Chinese (CTB) for the different models. We separate results for those that make do not make use of annotated data (top) versus those that do (mid). Note that our PTB setup from Dyer et al. (2016) differs considerably from the usual language modeling setup Mikolov et al. (2010) since we model each sentence independently and use a much larger vocabulary (see section 3.1).
['Model', 'PTB PPL', 'PTB  [ITALIC] F1', 'CTB PPL', 'CTB  [ITALIC] F1']
As a language model URNNG outperforms an RNNLM and is competitive with the supervised RNNG. The left branching baseline performs poorly, implying that the strong performance of URNNG/RNNG is not simply due to the additional depth afforded by the tree LSTM composition function (a left branching tree, which always performs reduce when possible, is the “deepest” model). The right branching baseline is essentially equivalent to an RNNLM and hence performs similarly. We found PRPN with default hyperparameters (which obtains a perplexity of 62.0 in the PTB setup from Mikolov et al. to not perform well, but tuning hyperparameters improves performance. The supervised RNNG performs well as a language model, despite being trained on the joint (rather than marginal) likelihood objective. This indicates that explicit modeling of syntax helps generalization even with richly-parameterized neural models. Encouraged by these observations, we also experiment with a hybrid approach where we train a supervised RNNG first and continue fine-tuning the model (including the inference network) on the URNNG objective This approach results in nontrivial perplexity improvements, and suggests that it is potentially possible to improve language models with supervision on parsed data. Note that we induce latent trees directly from words on the full dataset. For RNNG/URNNG we obtain the highest scoring tree from qϕ(z|x) through the Viterbi inside (i.e. CKY) algorithm. We calculate unlabeled F1 using evalb, which ignores punctuation and discards trivial spans (width-one and sentence spans). Since we compare F1 against the original, non-binarized trees (per convention), F1 scores of models using oracle binarized trees constitute the upper bounds.