Offline bilingual word vectors, orthogonal transformations and the inverted softmax
Table 6: “Translation” precision @1, when seeking to retrieve the true translation of an English sentence from a bag of 200k Italian sentences, or vice versa, averaged over 5k samples. We first obtain bilingual word vectors, using either the word dictionary provided by Dinu, or by constructing a phrase dictionary from Europarl. We set ns= 12800 in the inverted softmax.
['[EMPTY]', 'English to Italian: Word dictionary', 'English to Italian: Phrase dictionary', 'Italian to English: Word dictionary', 'Italian to English: Phrase dictionary']
Second, we can apply our orthogonal transformation to retrieve the Italian translation of an English sentence, or vice versa. To achieve this, we hold back the final 200k English and Italian sentences from our 500k sample of Europarl, and attempt to retrieve the true translation of a given sentence in this test set. We obtain the orthogonal transformation by performing the SVD on either the expert word dictionary provided by Dinu, or on a phrase dictionary formed from the first 300k sentences from Europarl. For simplicity, we do not apply dimensionality reduction here. For precision @1, most approaches favour the phrase dictionary, while Dinu’s method favours the word dictionary. We show in the appendix that all methods favour the phrase dictionary for precision @5 and @10. Remarkably, given no information except the sentence vectors, we are able to retrieve the correct translation of an English sentence with 67.8% precision. This is particularly surprising, since we are using the simplest possible sentence vectors, which have no information about word order or sentence length. When training the inverted softmax, the inverse temperature β diverged, and the “translation” performance from English to Italian significantly exceeded the performance from Italian to English. This improved the performance from Italian to English from 48.6% to 65.6%, which suggests that the optimal retrieval approach would be able to tune continuously between the conventional softmax and the inverted softmax. Once again, the inverted softmax performs well when retrieving the Italian translations of English sentences, but is less effective translating Italian sentences into English. However, the performance of Dinu’s method appears to rise more rapidly than other methods as we transition from precision @1 to @5 to @10. Additionally, while Dinu’s method performs better when using the word dictionary @1, it prefers the phrase dictionary @5 and @10.