On Learning Universal Representations Across Languages
Table 4: BLEU scores [%]. We conduct experiments with both low-resource and high-resource settings. Two bert-fused NMT models (Yang et al., 2020; Zhu et al., 2020) are considered as our baselines. Following (Gururangan et al., 2020), we also adopt pre-trained language models to downstream tasks by introducing a second phase of pre-training for Hictl on IWSLT or WMT parallel corpora, which denoted as Hictl∗. Results with ‡ are from our in-house implementation.
['Model', 'Low-Resource Iwslt’14', 'Low-Resource Iwslt’14', 'Low-Resource Iwslt’17', 'Low-Resource Iwslt’17', 'High-Resource Wmt’14', 'High-Resource Wmt’14', 'High-Resource Wmt’18', 'High-Resource Wmt’18']
We implemented standard Transformer (apply the base and big setting for IWSLT and WMT tasks respectively) as baseline. The proposed Hictl can improve the BLEU scores of the eight tasks by 1.93, 1.76, 2.7, 2.2, 1.43, 1.25, 1.87 and 1.67. Note we achieve new state-of-the-art results on IWSLT’14 En→De, IWSLT’17 En→{Fr, Zh} translations. These improvements show that mapping different languages into an universal representation space by using our Hictl to bridge the semantic discrepancy among cross-lingual sentence, is beneficial for both low-resource and high-resource translations.