Acknowledgements
Table 5.5: The results of various parent models. Each column specifies the size of parent training data, which is randomly downsampled from the original. Each row specifies parent model relatedness. The scores are in BLEU and specify performance of child model trained from the parent. For models with the star the performance dropped quickly during child training.
['[EMPTY]', '2M', '5M', '10M', '20M']
Surprisingly, the Spanish→Russian (with a target languages that uses the Cyrillic script) reached a better performance than the Spanish→French, a target language that is linguistically closest to the Estonian→English from all four investigated language pairs. However, neither of these two systems is significantly better than the other. Furthermore, the gains are quite similar (+0.49 up to +0.78 BLEU), which supports an assumption that the major factor influencing transfer learning is the size of the parent (here, all parents have 10M sentence pairs). This result can also be explained with a similar domain of parent training set. In comparison, the Czech→English parent, which has 40.1M sentences from a broader range of domains and has a shared language (English), improved the performance of Estonian→English by 3.38 BLEU. We can see that having more data can be more useful than a related language with fewer data. For example, 50% parent with 2M parallel sentences reached 16.76 BLEU, in contrast, having ten times more data but 0% related parent yields 18.25 BLEU. On the opposite, whenever the difference is only double, then the relatedness helps more with fewer data (19.13 BLEU vs. 18.25 BLEU). With 5M training data, the 50% parent already performs better than 20M with 0% (19.63 BLEU vs. 18.25). BiRNN Bidirectional Recurrent Neural Network BP Brevity Penalty BPE Byte Pair Encoding CNN Convolutional Neural Network IWSLT International Workshop on Spoken Language Translation LanideNN Language Identification by Neural Networks LM Language Model LSTM Long Short-Term Memory cells MT Machine Translation NLP Natural Language Processing NMT Neural Machine Translation NN Neural Network PBMT Phrase-Based Machine Translation POS Part-of-Speech OOV Out-of-Vocabulary RNN Recurrent Neural Network SMT Statistical Machine Translation T2T Tensor2tensor WAT Workshop on Asian Translation WMT Workshop on Statistical Machine Translation