HUME: Human UCCA-Based Evaluation of Machine Translation
Table 3: IAA for the multiply-annotated units, measured by Cohen’s Kappa.
['[EMPTY]', 'cs', 'de', 'pl', 'ro']
In order to assess the consistency of the annotation, we measure the Inter-Annotator Agreement (IAA) using Cohen’s Kappa on the multiply-annotated units. We report the overall Kappa, as well as separate Kappas on atomic units (annotated as Red, Orange or Green) and structural units (annotated as Adequate or Bad). This results in the Kappa for all units being considerably higher than the Kappa over the atomic units or structural units, where there is more internal confusion. We see no discernible reduction of IAA with sentence length. However, there are differences observed when we break down by node type. Specifically, we see a contrast between Czech and Polish, where the IAA is higher for atomic than for structural units, and German and Romanian, where the reverse is true. We also observe low IAA (around 0.3) in the cases of German atomic units, and Polish and Czech structural units.