Controlling the Output Length of Neural Machine Translation
Table 6: Manual evaluation on En-It (large data) ranking translation quality of the baseline (standard) and token short translation against the reference translation.
['[EMPTY]', '% of Wins', 'LR [ITALIC] src']
After manually inspecting the outputs of the best performing models under the large data condition, we decided to run a human evaluation only for the En-It Len-Tok model. As our ultimate goal is to be able to generate shorter translations and as close as possible to the length of the source sentences, we focused the manual evaluation on the Short output class and aimed to verify possible losses in quality with respect to the baseline system. We collected three judgments for each output, from 19 annotators, for a total of 807 scores (one sentence had to be discarded). Inter-annotator agreement measured with Fleiss’ kappa was 0.35 (= fair agreement).