Automatic Accuracy Prediction for AMR Parsing
Table 6: Results of different parse-ranking systems with respect to sentence-level parse rankings. ¯ρ: average Pearson-r on a sentence level. %pos: ratio of predicted rankings with positive ρ to gold ranking.
['[EMPTY]', 'Smatch LDC2015E86 ¯ [ITALIC] ρ', 'Smatch LDC2015E86 %pos', 'Smatch BioAMRTest ¯ [ITALIC] ρ', 'Smatch BioAMRTest %pos']
When only choosing among CAMR and JAMR outputs, on LDC2015E86, our system boosts the F1 by 2.7 pp. compared to randomly selecting a parse, and by 0.6 pp. compared to always choosing the parse from the better system (determined on dev, here: JAMR). Choosing from CAMR and GPLA or JAMR and GPLA makes little difference: in most cases our system selects the GPLA parse and the difference to only choosing GPLA parses is marginal. In sum, we can draw two conclusions from this experiment: given a sentence, ranking AMR parser outputs using our accuracy prediction model, on in-domain and out-of-domain unseen data (i) clearly improves performance when non state-of-the-art parsers are applied or if we are not informed about the parsers’ performances and (ii) does not worsen results in other cases.