All Word Embeddings from One Embedding
Table 2: Results of WMT En-De translation.
['Method', '[ITALIC] Dinter', 'Embed', 'BLEU']
In addition, we trained the Transformer with a simple factorization approach as a baseline. In the simple factorization approach, we construct word embeddings from one small embedding matrix and weight matrix to expand a small embedding to one with the larger dimensionality De. We modified the dimension size to make the number of parameters related to embeddings almost equal to those in ALONE. This result indicates that our ALONE can reduce the number of parameters related to embeddings without negatively affecting the performance on machine translation. We cannot demonstrate the embedding parameter size for Transformer+DeFINE because its vocabulary size is unreported but the parameter size of Transformer+DeFINE is 68M, which is larger than that of the original Transformer (60.9M). We consider that the original Transformer (and our experiments) save the total parameter size by sharing embeddings with the pre-softmax linear transformation matrix.