Normalization of Input-output Shared Embeddings in Text Generation Models
Table 3: BLEUs for Transformer models on IWSLT 14’ datasets with different embedding sharing techniques.
['Task', 'Configuration', 'BLEU']
For Transformer models on IWSLT 14’ translation tasks, the 4 new weight sharing methods also outperforms the baseline. In this circumstance, square-normalized output embedding method is the best weight sharing method among all methods, which achieves BLEU improvements of 0.86/0.73/0.60 on English to German/Spanish/Romania translation tasks. The Transformer model for IWSLT 14’ translation tasks has smaller embedding size. As making the total embedding matrix normalized loses some degrees of freedom for the data of embedding vectors, it will have greater impact on embedding spaces with lower dimensions. Therefore, the method of l2-normalized input embedding can not perform as well as in WMT 16’ dataset.