Exploring Transformers for Large-Scale Speech Recognition
Table 4: Results of deeper Transformer models. L denotes the model depth.
['Model', 'IC', 'Size(M)', '[ITALIC] L', 'Context', 'dev', 'eval']
We observe similar trend on the eval set. We also investigated the tradeoff between increasing the dimension of hidden state in self-attention layers and increasing the depth the model. For the offline 12-layer Transformer, we increased dk to 960, which resulted in a model with 97 million parameters. However, we only achieved very marginal improvement on the dev set. The gain from increasing the model depth to 24 layers is more considerable on the dev set, but the gain on the eval set is still small. It is possible that the model is overfitted, and increasing the dropout ratio may result in more gains. As for Transformer-XL, we can obtain gains in terms of accuracy by increasing the depth up to 48 layers. However, the gains are not as large as we have expected, and regularizing the deep Transformers may result in further improvements.