Improving Neural Parsing by Disentangling Model Combination and Reranking Effects
Table 2: Development F1 scores on section 22 of the PTB when using various models to produce candidates and to score them. ∪ denotes taking the union of candidates from each of two models; + denotes using a weighted average of the models’ log-probabilities.
['Candidates', 'Scoring models RD', 'Scoring models LM', 'Scoring models RD + LM']
Each row is a different candidate set, where the third row in each table presents results for the augmented candidate sets; each column is a different scoring model, where the third column is the score combination setting described below. Going from RD → RG to the augmented candidate setting RD ∪ RG → RG decreases performance from 93.45 F1 to 92.78 F1 on the development set. This difference is statistically significant at the p<0.05 level under a paired bootstrap test. We see a smaller, but still significant, effect in the case of LM: RD → LM achieves 93.66, compared to 93.47 for RD ∪ LM → LM. We find that combining the scores of both models improves on using the score of either model alone, regardless of the source of candidates. These improvements are statistically significant in all cases. Score combination also more than compensates for the decrease in performance we saw previously when adding in candidates from the generative model: RD ∪ RG → RD + RG improves upon both RD → RG and RD ∪ RG → RG, and the same effect holds for LM.