Automatic Accuracy Prediction for AMR Parsing
Table 7: True rank r (given corpus-Smatch) and predicted rank ^r (based on sentence average Smatch computed using our model). p1: probability of non-correlation. p2: probability that a randomly produced ranking achieves equal or greater ρ (estimated over 106 random rankings). For team names, see fn. 5 & 6.
['[EMPTY]', 'Rank LDC2015R36 rank  [ITALIC] r', 'Rank LDC2015R36 rank ^ [ITALIC] r', 'Rank BioAMRTest rank  [ITALIC] r', 'Rank BioAMRTest rank ^ [ITALIC] r']
S4SS3SSS0Px1 Results On BioAMRTest we have a good, albeit non statistically significant correlation with the true team ranking. On the in-domain LDC2015R36 test set we see a significant correlation of ρ=0.645 (p1,2<0.05). In this shared task, many teams were competitive and differences between the best teams were marginal. For example, in the true ranking, places 1 to 6 achieved between 0.60 and 0.62 Smatch F1. Notably, the first four teams according to the true ranking and the first four teams according to our predicted ranking fall into the same group. This shows that our model successfully assigned high ranks to low error submissions.