Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation
Table 3: Accuracies on the test set of the MLDoc dataset. *mBERT numbers have been taken from wu2019beto. mBERT is also the state-of-the-art for in-language training. † For zero-shot SOTA refers to artetxe2018massively.
['[EMPTY]', 'In language mBERT*', 'In language MMTE', 'Zero-shot mBERT*', 'Zero-shot MMTE', 'Zero-shot SOTA†']
MLDoc is a balanced subset of the Reuters corpus covering 8 languages for document classification (SCHWENK18.658). This is a 4-way classification task of identifying topics between CCAT (Corporate/Industrial), ECAT (Economics), GCAT (Government/Social), and MCAT (Markets). Performance is evaluated based on classification accuracy. We split the document using the sentence-piece model and feed the first 200 tokens into the encoder for classification. The task-specific network and the optimizer used is same as the one used for XNLI. Learning rate schedule is (0.2,5k). We perform both in-language and zero-shot evaluation. The in-language setting has training, development and test sets from the language. In the zero-shot setting, the train and dev sets contain only English examples but we test on all the languages.