On the Evaluation of Semantic Phenomena inNeural Machine Translation Using Natural Language Inference
Cuadro 8: Accuracies on FN+’s dev set based on sentence length. The first column represents the range of sentences length: first number is inclusive and second is exclusive. The last column represents how many context sentences have lengths that are in the given row’s range.
['Sentence length', 'ar', 'es', 'zh', 'de', 'total']
When trained on sentence representations form an English→Chinese,German NMT encoder, the NLI accuracies steadily decrease. When using English→Arabic, the accuracies stay consistent until sentences have between 70–80 tokens while the results from English→Spanish quickly drops from 0–10 to 10–20 but then stays relatively consistent.