Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard-300
Table 4: Reducing the model size
['enc. depth', 'enc. LSTM', 'enc. lin.', 'dec. LSTM', '#par.', 'WER w/o LM', 'WER w/o LM', 'WER w/ LM', 'WER w/ LM']
A 280M-parameter model may be larger than is practical in many applications. Thus, we also conduct experiments to see if this model size is necessary for reasonable ASR performance. Models are trained without changing the training configuration, except that the size or number of LSTM layers is reduced. Nevertheless, an external language model reduces the performance gap. A small, 57M-parameter model together with a similar size language model is only 5% relative worse than our largest model.