Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation
Table 4: Accuracies on the test set of the intent classification dataset. *mBERT numbers are from our own implementation using the publicly available mBERT checkpoint. † SOTA refers to the numbers reported in schuster2018cross.
['[EMPTY]', 'In language mBERT*', 'In language MMTE', 'In language SOTA†', 'Zero-shot mBERT*', 'Zero-shot MMTE', 'Zero-shot SOTA†']
schuster2018cross recently presented a dataset for multilingual task oriented dialog. This dataset contains 57k annotated utterances in English (43k), Spanish (8.6k), and Thai (5k) with 12 different intents across the domains weather, alarm, and reminder. The evaluation metric used is classification accuracy. We use this data for both in-language training and zero-shot transfer. The task-specific network and the optimizer used is the same as the one used for the above two tasks. The learning rate schedule is (0.1,100k). MMTE outperforms both mBERT and previous SOTA in both in-language and zero-shot setting on all 3 languages and establishes a new SOTA for this dataset.