Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
Table 2: Effect of switching encoders on WER
['Random Seed', 'Loss Criterion CTC', 'Loss Criterion CE', 'Eval2000 SWB', 'Eval2000 CH']
The decoder cross-attention over the encoder hidden representation makes it not only conditioned on the encoder outputs but also dependent on the encoder architectural decisions and internal hidden representations. The whole ASR system fall apart under the interchangability test, i.e. switching an encoder with another similar one that is only different in its initial random seed, which brings our point about the lack of modularity in encoder-decoder models.