AMR Normalization for Fairer Evaluation
Table 2: Difference in Smatch score with and without reification (top is gold, rest are test, bold are differences)
['AMR', 'Collapsed', 'Reified']
The effect of reification on the Smatch score can be large. By reifying one relation to a node with two relations, the net total of triples increases by two. The result is that Smatch will require more time and memory to compute a score, and the search for the variable mapping may become less stable because there are more nodes to search over. This normalization can affect the Smatch score by amplifying certain kinds of errors and giving partial credit for others. The Collapsed column shows the Smatch score between the gold and test AMRs when the relations are left as-is, and the Reified column shows the score when both gold and test are reified.