Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
(a) The target languages in this one-to-many translation task belong to the same language family. Ro and Fr are Romance languages while De and Nl are Germanic languages.
['[ITALIC] Set of shared parameters (Θ)', 'En→De+Tr →De', 'En→De+Tr →Tr', 'En→De+Ja →De', 'En→De+Ja →Ja', '[ITALIC] params ×106']
Here, we first analyze the results of one-to-many multilingual translation experiments when there are two target languages and both of them belong to the same language family. The first set of experiments are on Romance languages (En→Ro+Fr) and the second set of experiments are on Germanic languages (En→De+Nl). We observe that sharing only the embedding layer weight between the multiple models leads to the lowest scores. Sharing the encoder weights results in significant improvement for En→Ro+Fr but leads to a small decrease in En→De+Nl scores. Next, we analyze the results of one-to-many translation experiments when both the target languages belong to distant language families and are unrelated. The first set of experiments are on Germanic, Turkic languages (En→De+Tr) and the second set of experiments are on Germanic, Japonic languages ( En→De+Ja). Here, we observe that the approach of sharing all the parameters leads to a noticeable drop in the BLEU scores for both the considered language pairs. Similar to the above discussion, sharing the key and query matrices results in a large increase in the BLEU scores. We hypothesize that in this partial parameter sharing strategy, the sharing of key and query attention weights effectively models the common linguistic properties while the separate FFN sublayer weights model the unique characteristics of each target language, thus overall leading to a large improvement in the BLEU scores. The results of other decoder parameter sharing approaches lie close to the key and query parameter sharing method. As the target languages are from different families, their vocabularies may have some overlap but will be significantly different from each other. In this scenario, a useful alternative is to consider a separate embedding layer for every source-target language pair while sharing all the encoder and decoder parameters. However, we did not experiment with this approach, as the inclusion of separate embedding layers will lead to a large increase in the model parameters and as a result model training will become more memory intensive. We leave the investigation of such parameter sharing strategy to future work.