AMR Similarity Metrics from Principles
Table 1: Symmetry violation ratio (Eq. 4) (in %); mean symmetry violation (Eq. 5) of AMR metrics.
['[EMPTY]', 'symmetry violation svr (%, Δ>0.0001)', 'symmetry violation svr (%, Δ>0.0001)', 'symmetry violation msv (in pp.)', 'symmetry violation msv (in pp.)']
We perform the experiment with parses from three systems: CAMR wang2016camr, GPLA lyu-titov-2018-amr and JAMR flanigan-etal-2014-discriminative and the gold graphs, and compare the ratio and mean symmetry violation of Smatch and SemBleu (cf. Specifically, we fetch 16 system outputs of the WMT 2018 en-de metrics task DBLP: conf/wmt/MaBG18 and calculate Bleu(A,B) and Bleu(B,A) based on the MT system’s output and the reference sentences (using the same smoothing method as SemBleu). As worst-case/avg.-case, we use the outputs from the team where Bleu exhibits maximum/median msv. The average mean symmetry violation of Smatch is considerably smaller compared to SemBleu: 0.1 vs. 3.2 percentage points (pp.) F1 score. Even though the Bleu metric is inherently asymmetric (which is reflected by a high svr-ratio), most of the symmetry violations are negligible when applied in MT. However, the Bleu metric applied to AMR graphs with SemBleu amplifies Bleu’s asymmetry by a factor of approximately 16 (0.2 pp. vs. 3.2 pp. The Sembleu-plots show that the effect is widespread, some cases are extreme, many others are less extreme but still considerable.