Handling Homographs in Neural Machine Translation
Table 3: Translation results for homographs and all words in our NMT vocabulary. We compare scores for baseline and our best proposed model on three different language pairs. Improvements are in italic. We performed bootstrap resampling for 1000 times: our best model improved more on homographs than all words in terms of either f1, precision, or recall with p<0.05, indicating statistical significance across all measures.
['language', 'System', 'Homograph F1', 'Homograph Precision', 'Homograph Recall', 'All Words F1', 'All Words Precision', 'All Words Recall']
To do so, we first acquire the translation of homographs in the source language using fast-align Dyer et al. The settings follow the default command on fast-align github page including heuristics combining forward and backward alignment. Since there might be multiple aligned words in the target language given a word in source language, we treat a match between the aligned translation of a targeted word of the reference and the translation of a given model as true positives and use F1, precision, and recall as our metrics, and take the micro-average across all the sentence pairs. We calculated the scores for the 50000 words/characters from our source vocabulary using only English words. The table shows two interesting results: (1) The score for the homographs is lower than the score obtained from all the words in the vocabulary. This shows that words with more meanings are harder to translate with Chinese as the only exception. (2) The improvement of our proposed model over baseline model is larger on the homographs compared to all the words in vocabulary. This shows that although our context-aware model is better overall, the improvements are particularly focused on words with multiple senses, which matches the intuition behind the design of the model. We hope that this paper will spark discussion on the topic, and future work will propose even more focused architectures.