Lessons learned in multilingual grounded language learning
Table 1: Vocabulary overlap as measured by the Jaccard coefficient between the different languages on the translation portion of the Multi30K dataset.
['[EMPTY]', 'En', 'De', 'Fr', 'Cz']
We train and evaluate our models on the translation and comparable portions of the Multi30K dataset elliott2016multi30k; elliott2017findings. The translation portion (a low-resource dataset) contains 29K images, each described in one English caption with German, French, and Czech translations. The comparable portion (a higher-resource dataset) contains the same 29K images paired with five English and five German descriptions collected independently. To reduce the vocabulary size of the joint models, we replace all words occurring fewer than four times with a special “UNK” symbol. The total number of tokens across all four languages is 17,571, and taking the union of the tokens in these four languages results in vocabulary of 16,553 tokens – a 6% reduction in vocabulary size. On the comparable portion of the dataset, the total vocabulary between English and German contains 18,337 tokens, with a union of 17,667, which is a 4% reduction in vocabulary size.