Recurrent Stacking of Layers for Compact Neural Machine Translation Models
Table 1: Results for our experiments on WMT, ALT, KFTT and the GCP data-sets. The last row denoted as ‘6-layer model‘ is a transformer model with 6-layers without any parameter sharing. The results in bold indicate the best values among the diverse models of the proposed method.
['[BOLD] #recurrently stacked layers', '[BOLD] BLEU  [BOLD] WMT', '[BOLD] BLEU  [BOLD] WMT', '[BOLD] BLEU  [BOLD] ALT', '[BOLD] BLEU  [BOLD] KFTT', '[BOLD] BLEU  [BOLD] GCP', '[BOLD] BLEU  [BOLD] GCP']
We observed that, no matter the data-set, the translation quality improves as the same parameters are recurrently used in a depth-wise fashion. The most surprising result is that the performance of our 6-layer recurrently stacked model with shared parameters across all layers approaches the performance of the vanilla 6-layer model without any parameter sharing across layers. The most probable explanation of this phenomenon is that the parameter sharing forces the higher layers of the NMT model to learn more complex features. Note that dropout is applied by default in the transformer implementation we use. Thus, at each stage, the same set of parameters has to make do with less reliable representations. This means that the representations at the topmost layers are very robust and thus enable better translation quality. The gains in translation quality are inversely proportional to the amount of recurrent stacking layers.