An Empirical Analysis of NMT-Derived Interlingual Embeddings and their Use in Parallel Sentence Identification
Table V: Comparison of the Pearson correlation obtained by context vectors at different epochs of the training and word embeddings on the test set of the “Semantic Textual Similarity Task’’ at SemEval 2017.
['[EMPTY]', 'track1', 'track2', 'track3', 'track4a', 'track5']
Observing the evolution through training by taking a shot at four different points, the correlation increases with the number of iterations for all the language pairs and systems. In this fine-grained task, the internal representation improves in parallel to the translation quality. As before, the system with words is better than the one with lemmas with the only exception of ar–en. A reason could be the low initial similarity for semantically equivalent sentences (trad) for this pair with the S1-w system (0.26±0.10). The initial point seems to be relevant for the final performance; i.e. the relative improvement from epoch to epoch for all language pairs is very similar, but the final performance seems to be conditioned to the quality of the initial representations. The performance in the monolingual tracks is always higher than in the cross-language ones, and the difference at the end of the training is proportional to the difference at the beginning. The study of how a proper initialisation of the input word embeddings could alleviate this disparity deserves future research. We estimated 300 (WE-d300-nmt) and 1024 (WE-d1024-nmt) dimensional word embeddings with the same corpus used to train the NMT systems (adding monolingual corpora did not improve the results). When sentences belong to different languages, we translate them into en and use the embeddings estimated for en. As done with context vectors, the similarity between sentences is assessed by the cosine of the summed embeddings. Higher-dimensional word embeddings outperform the 300-dimensional ones in the task.