Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard-300
Table 3: Optimizing dropout (dropo.), DropConnect (dropc.), layer and bottleneck (bn) size for LSTM LM, optionally modeling across utterances (x-utt.)
['model dropo.', 'model dropc.', 'model width', 'model bn', 'model x-utt.', 'PPLword CV', 'PPLword Hub5’00', 'WER swb', 'WER chm']
The following experiments summarize our optimization of the LM. Increasing the model capacity to 122M parameters results in a significant gain in PPL only after the dropout rates are tuned (rows 3, 5 and 6). However, this PPL improvement does not translate into reduced WER with a bigger model when cross utterance modeling is used (rows 4 and 7). Thus, in all other experiments we use the smaller, 57M-parameter model.