Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks
Table 1: Impact of training regime and vocabulary overlap on agreement accuracy (FR→IT).
['[EMPTY]', 'IT [ITALIC] small', 'Bilingual (FR+IT [ITALIC] small) Joint Training', 'Bilingual (FR+IT [ITALIC] small) Joint Training', 'Bilingual (FR+IT [ITALIC] small) Pre-Training', 'Bilingual (FR+IT [ITALIC] small) Pre-Training', 'IT [ITALIC] large']
Pre-training considerably reduces the negative interference effect observed in joint training, and even leads to a higher accuracy on Original sentences in the no-overlap setup (83.2 vs 79.8). Eliminating vocabulary overlap (None) also leads to better agreement scores in most cases. The best gain overall is obtained by the jointly trained model with no overlap (85.7 vs 79.8) in the Original sentences, whereas no gain is observed in the Nonce sentences.