Grammar Induction with Neural Language Models:An Unusual Replication
Table 3: Unlabeled parsing F1 on the MultiNLI development set for models trained on AllNLI. F1 wrt. shows F1 with respect to strictly right- and left-branching (LB/RB) trees and with respect to the Stanford Parser (SP) trees supplied with the corpus; The evaluations of SPINN, RL-SPINN, and ST-Gumbel are from Williams et al. (2018a). SPINN is a supervised parsing model, and the others are latent tree models. Median F1 of each model trained with 5 different random seeds is reported.
['[BOLD] Model', '[BOLD] Stopping  [BOLD] Criterion', '[BOLD] F1 wrt.  [BOLD] LB', '[BOLD] F1 wrt.  [BOLD] RB', '[BOLD] F1 wrt.  [BOLD] SP', '[BOLD] Depth']
We conclude that PRPN does acquire some substantial knowledge of syntax, and that this knowledge agrees with Penn Treebank (PTB) grammar significantly better than chance.