Effective Inference for Generative Neural Parsing
Table 5: Comparison of F1 scores on section 23 of the Penn Treebank. Here we only include models trained without external silver training data. Results in the first two sections are for single-model systems.
['Parser', 'LR', 'LP', 'F1']
We find that the best overall settings are a beam size of k=2000, a word beam size of kw=200, and ks=20 fast-track candidates per step, as this setup achieves both the highest probabilities under the model and the highest development F1. We achieve F1 scores of 92.56 on the test set without pruning and 92.53 when 1−8/26≈69.2% of Open successors are pruned, obtaining performance well above the previous state-of-the-art scores for single-model parsers. The fact that we match the performance of their reranking parser using the same generative model confirms the efficacy of our approach. We believe that further refinements of our search procedure can continue to push the bar higher, such as the use of a learned heuristic function for forward score estimation, or a more sophisticated approximate decoding scheme making use of specific properties of the model. We look forward to exploring these directions in future work.