Transfer Learning for Low-Resource Neural Machine Translation
Table 7: Starting with the parent French-English model (Bleu =24.4, PPL=6.2), we randomly assign Uzbek word types to French word embeddings, freeze various parameters of the neural network model, and allow Uzbek-English (child model) training to modify other parts. The table shows how Uzbek-English Bleu and perplexity vary as we allow more parameters to be re-trained.
['Setting', 'Dev Bleu', 'Dev PPL']
In all the above experiments, only the target input and output embeddings are fixed during training. In this section we analyze what happens when different parts of the model are fixed, in order to see what yields optimal performance. We see that the optimal setting for transferring from French-English to Uzbek-English in terms of Bleu performance is to allow all of the components of the child model to be trained except for the input and output target embeddings.