Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax
Table 1: Precision at N (P@N) (%) of target sentence retrieval on the UN corpus. Models attempt to select the true translation target for a source sentence from the entire corpus (11.3 million aligned sentence pairs). [Guo et al.2018] is a dual encoder (DE) model trained with deep averaging network (DAN) instead of transformer. In the last row, AM stands for “Additive Margin softmax”.
['Models', 'en-fr P@1', 'en-fr P@3', 'en-fr P@10', 'en-es P@1', 'en-es P@3', 'en-es P@10', 'en-ru P@1', 'en-ru P@3', 'en-ru P@10', 'en-zh P@1', 'en-zh P@3', 'en-zh P@10']
The baseline systems are listed in the first two rows. Rows 3 to 5 show the results from our models: a unidirectional dual-encoder (DE) model, a bi-directional dual-encoder (BiDE) model, and the bi-directional model with additive margin softmax (BiDE+AM). The BiDE is slightly but consistently better than DE for all four language pairs. BiDE+AM further improves performance over BiDE, with a minimum of 86.1% P@1 in all language pairs. BiDE+AM greatly outperforms the baseline models in both en-fr and en-es, and establishes a new state-of-the-art in performance for this task.