Automatic Accuracy Prediction for AMR Parsing
Table 4: Various percentiles of Smatch F1 predictions for gold graphs.
['dataset', 'percentile 5', 'percentile 25', 'percentile 75', 'percentile 90', 'percentile 95', 'percentile 97', 'percentile 99']
Evidently, our system also gets the ranking of the parsing systems right: the distribution of the state-of-the-art (GPLA) is shifted right towards higher predicted F1 scores, whereas the distribution of CAMR is shifted left towards lower scores. Nevertheless, the prediction score density for gold parses is situated more on the right hand side than most others. In other words, we find that in the out-of-domain data gold parses tend to be assigned above-average scores.