Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
Table 5: WER without (scratched) and with modular upgrade of the decoupled model.
['Architecture', 'RF', 'Top k', 'Original Enc SWB', 'Original Enc CH', 'Decoder SWB', 'Decoder CH']
Another benefit of modular independence, which we enjoy in software design but not in building fully-differentiable seq2seq models, is the ability to carefully build one critical module to higher levels of performance then switch it into the full system without the need for any model fine-tuning. The new module may reflect a new architecture design in that module, e.g. from LSTMs to Transformers, or simply more training data that become available for that module. In our case, we just used an encoder model that is trained independently using the CTC loss for a larger number of updates.