An Empirical Analysis of NMT-Derived Interlingual Embeddings and their Use in Parallel Sentence Identification
Table VI: Precision, recall and F1 scores on the binary classification of pseudo-alignments on the held-out test set.
['[EMPTY]', '[EMPTY]', '[ITALIC] de– [ITALIC] en P', '[ITALIC] de– [ITALIC] en R', '[ITALIC] de– [ITALIC] en F1', '[ITALIC] fr– [ITALIC] en P', '[ITALIC] fr– [ITALIC] en R', '[ITALIC] fr– [ITALIC] en F1', 'joint P', 'joint R', 'joint F1']
Notice that a greedy threshold search is better than any of the machine learning counterparts when only context vectors are used, but differences are not significant. The greedy search on the context vector similarities gives a better F1 on the held-out test set than an ensemble of SVM and GB operating only the set of additional features with almost no knowledge of semantics. As we argued in the previous section, translations and non-translations are clearly differentiated by a cosine similarity of the context vectors for these languages pairs, as the difference between the mean similarities of translations and unrelated texts is much higher than its uncertainty (Δtr−ur=0.36±0.14 for de–en, and 0.41±0.14 for fr–en). This clear distinction in the similarities is translated into an F1=98.2% in the task of parallel sentence identification. By contrast, the set of complementary features depends on the language pair and shows a performance drop for de–en. For this reason, the results in the multilingual setting are always worse than in the bilingual one. This fact is inherited in the all scenario, where the classification for the joint corpus obtains F1=98.9%, which is lower than the one obtained for fr–en alone (F1=99.3%). Nevertheless, semantic and syntactic similarity features are complementary and the combination of all similarity measures slightly improves precision, recall and F1 in the multilingual setting. It is worth noting the high recall derived from the context vectors, which reaches 100% for fr–en and falls to 98.1% for the joint data, being still 6.5 points higher than for the comp features.