Meta-Embeddings Based On Self-Attention
Table 6: Ablation on WMT En-De validation set. (+) indicates that a result includes all preceding features.
['Model', 'Param', 'BLEU']
In order to evaluate the function of different parts in our architecture, we did an ablation test on the WMT 2014 En-De Validation set. Initially, we add the meta embeddings to the Vanilla Transformer Model, and it seems that this gives the most salient advancement of the performance. However, the number of parameters increased quite a lot, and the improvement may merely come from the additional parameter numbers. Therefore, we decided to shrink the model’s size by using weight sharing in Duo Multihead. It turns out that this operation not only reduced the number of parameters but also improves the performance. The following Normalization and Fusion layer has also been proved to be beneficial.