Multi-Layer Softmaxing during Training Neural Machine Translationfor Flexible Decoding with Fewer Layers
Table 1: BLEU scores for the WMT En→De task. The scores on the left side are for the 36 individual models that are trained separately. The scores on the right are for our proposed N×M model.
['[ITALIC] n∖ [ITALIC] m', '1', '2', '3', '4', '5', '6']
These summarize the cost-benefit property of our N×M model in comparison with the results of the corresponding vanilla models. When our N×M model was used for decoding with the 5 encoder and 5 decoder layers, it achieved a BLEU score of 34.95 which is comparable with the BLEU score of 35.35 of the best vanilla model with 6-layer encoder and 4-layer decoder, even though the objective function for our proposed model is substantially more complex than the one for the vanilla model. Note that the vanilla models give significantly better results compared to our N×M models, when using a single decoder layer. However, when the number of decoder layers are increased there is no statistically significant difference between the performance of vanilla models and our N×M model; difference is less than 1.0 BLEU points in most configurations. We have essentially compressed 36 models into one. One may argue that training a single vanilla model with optimal numbers of encoder and decoder layers is enough. Recall that we aim at a flexible model and