AMR quality rating with a lightweight CNN
Table 7: Right column: results of our system when we abstain from feeding the dependency tree, and only show the sentence together with the candidate AMR.
['[EMPTY]', 'Quality Dim.', 'LG-LSTM', 'ours', 'ours (no dep.)']
The performance drops, compared with our full system, are rather small. However, the drops are consistent over all analyzed Smatch scores, both in terms of error (0 to 2.2% increase) and Pearson’s ρ (1.4 to 2.4% decrease). This indicates that the dependency trees contains information that can be exploited by our model to better judge the AMR quality. We hypothesize that this is due to similarities between relations such as subj/obj (syntactic) or arg0/arg1 (semantic), etc. However, we see that this simpler model still outperforms the LG-LSTM baseline, except in Smatch precision error, which is increased by 2.4%.