Automatic Accuracy Prediction for AMR Parsing
Table 5: Results (sentence averages) of different AMR parsing (bottom part) and ranking (top part) systems on two test sets. Upper part: results when selecting from alternative parses: lower-bound (upper-bound): oracle selecting the worst (best) AMR parse; ours: results when selecting the best parse according to our models’ accuracy prediction (hierarchical model).
['[EMPTY]', 'Smatch LDC2015E86 P', 'Smatch LDC2015E86 R', 'Smatch LDC2015E86 F1', 'Smatch BioAMRTest P', 'Smatch BioAMRTest R', 'Smatch BioAMRTest F1']
For our in-domain test data, LDC2015E86, selecting the best parse according to our model’s predicted accuracy score improves over all individual parser results: the obtained average Smatch F1 per sentence increases (i) slightly by 0.2 pp. compared to always choosing outputs from GPLA and (ii) observably by 5.7 pp. compared to randomly selecting a parse from the competing system outputs. The difference compared to always choosing GPLA seems negligible which perhaps can be explained by the fact that GPLA has been shown to be on par or better than doubly-blind human annotators. Smatch F1 increase compared to our model. This margin is small and further success might also be hampered by peculiarities in the manual annotations. On BioAMRTest, no prior information about the systems is available. Using our model’s predicted scores to select from the alternative system outputs, we can boost Smatch F1 by 5.2 pp. compared to randomly selecting a parse. Compared to always selecting the parses of the best submitted system (in-hindsight), we lag behind by 3.9 pp.