Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
Table 6: PostEdit conditional training of the Decoder module
['Architecture', 'RF', 'Top k', 'Encoder SWB', 'Encoder CH', 'PostEdit Dec. SWB', 'PostEdit Dec. CH']
The beam convolution attention preparation architecture, which uses only the rank of encoder hypotheses rather than probability values, shows much more resilience and ability to fix frozen parent module errors compared to the weighed embedding architecture. There is still a slight degradation of the final decoder performance when trained conditionally on the encoder output – without joint fine-tuning. The reason for that is the lack of data augmentation effect when training the decoder module, as a side effect of modular components, because the encoder is trained to be invariant to augmentation when producing its final probability distribution. This can be treated by designing data augmentation techniques suitable to be applied at the input of each module, which we refer to future work.