SemBleu: A Robust Metric for AMR Parsing Evaluation
Table 3: Sentence-level accuracies, where the highest n-gram order is set to 3 by default, unless specified.
['Metric', 'Percent (%)']
This indicates that SemBleu is more consistent with human judges than Smatch. The lower group shows SemBleu accuracies with different order n. With only unigram features (node-to-node correspondences), SemBleu is much worse than Smatch. When incorporating bigrams and trigrams, SemBleu gives consistently better numbers, demonstrating the usefulness of high-order features. Further increasing n leads to a decrease of accuracy. This is likely because humans care more about the whole-graph quality than occasional high-order matches.