On the Evaluation of Semantic Phenomena inNeural Machine Translation Using Natural Language Inference
Cuadro 3: Accuracies on FN+’s dev set based on whether the swapped paraphrases share the same POS tag.
['[EMPTY]', 'ar', 'es', 'zh', 'de']
Upon manual inspection, we noticed that in many not-entailed examples, swapped paraphrases had different part-of-speech (POS) tags. This begs the question of whether different POS tags for swapped paraphrases affects the accuracies. Using Stanford CoreNLP Manning et al. , we partition our validation set based on whether the paraphrases share the same POS tag. Classifiers using features from NMT encoders trained on the three languages from the UN corpus noticeably perform better on cases where paraphrases have different POS tags compared to paraphrases with the same POS tags. These differences might suggest that the recast FN+ might not be an ideal dataset to test how well NMT encoders capture paraphrastic inference. The sentence representations may be impacted more by ungrammaticality caused by different POS tags as opposed to poor paraphrases.