Language Modeling with Deep Transformers
Table 8: WERs (%) for attention-based models on LibriSpeech 960hr dataset. Perplexities are on the 10K BPE level.
['LM', 'Beam', 'dev clean', 'dev clean', 'dev other', 'dev other', 'test clean', 'test clean', 'test other', 'test other']
The 10K BPE level training data has a longer average length of 24 tokens per sentence with the longest sentence length of 1343, which is still manageable without any truncation for self-attention. We use the Transformer architecture of (24, 4096, 1024, 8). The LSTM model has 4 layers with 2048 nodes. Again, we obtain consistent improvements over the LSTM baseline. These results are better than previously reported WERs