Layer-Wise Cross-View Decoding for Sequence-to-Sequence Learning
Table 3: Results of different cross-view strategies on the IWSLT DE-EN dataset in terms of BLEU.
['Methods', 'Transformer', 'DynamicConv']
Analysis on Cross-View Decoding Strategies As we can see, all the considered strategies can improve the performance and GCA shows the best improvement in accuracy. GPA could not provide the fine-grained representations for the last decoder layer. For FGA, it only provides a single view of the source sequence. FMA combines all views and may introduce redundant noise, while the AMA strategy has a hard time in learning proper weights for different views. Those results further demonstrate the GCA strategy is more in line with the characteristics of the multi-layer encoding and decoding process and the information is provided more efficiently. Ablation Study In this section, we conduct the ablation study to investigate the contribution of each mitigation in the proposal. As we can see, without the mitigations, the accuracy is affected negatively when cross-view decoding is applied. In addition, between the two mitgations, continued learning is much more effective; but only if soft integration and continued learning are both implemented, we can observe a significant promotion in accuracy over the baseline. The reason is that for layer-wise cross-view decoding to work, each encoder layer must be able to produce representations with different granularity or characteristics. It can hardly be done if the model is trained from the beginning with cross-view decoding, since even with soft integration, GCA will allow error signal to bypass the top encoder layers. We also check if the accuracy improvement is simply due to longer training that comes with continued learning. If the conventional models are allowed the same more training time, their results are not bettered meaningfully. The increase is 0.0, 0.1, and 0.3 in terms of BLEU for DE-EN, EN-DE, and EN-FR, respectively. Overall, the two mitigations are both essential to the layer-wise cross-view decoding approach and result in a fully-augmentative training procedure that can be easily integrated into existing systems.