Exploring Transformers for Large-Scale Speech Recognition
Table 3: Results of streaming Transformer models. The number of layers is 12.
['Model', 'IC', 'Size (M)', '[ITALIC] N', '[ITALIC] dk', 'Context', 'dev']
The previous experiments focused on the offline scenario. In this section, we evaluate the accuracy of Transformers in the steaming condition. For instance, [−∞,40] corresponds to looking ahead 3 frames for each self-attention layer in a 12-layer Transformer without interleaved convolution, with an additional 4 frames latency from the VGG encoder. Since our Transformers operated at the 20 ms frame rate, 40 frames correspond to 800 ms latency. For the attention mask based Transformer, we did not limit the left context, so it is marked as −∞, while the context window as [−∞,∞] refers to the offline system. For Transformer-XL, we set the chunk size as 40, and since the model takes the hidden states from the previous chunk as feature, we denote the context size as [−40,40]. Note that there are not overlaps in the chunks during both training and inference, and it emits 40 outputs each time during inference. For LC-BLSTM, the chunk size is also 40, and because it takes the previous hidden state as the representation of history, arguably, we denote the context size as [−1,40]. The chunks in LC-BLSTM are overlapped by 20 frames, so it only emits 20 outputs each time during inference. First, without the interleaved convolution, the streaming Transformers based on attention masks degrade the recognition accuracy by 18% - 25% compared to the offline baseline model in the low latency condition. With the interleaved convolution, the accuracy loss is much smaller. When the latency constraint is less tight, the effect of interleaved convolution layers is diminishing. Second, Transformer-XL still lags behind the vanilla Transformer with the context window of [−∞,40]. This is not surprising as in Transformer-XL, the previous chunk is only an approximation of the full history. Third, the gap between the streaming Transformer (or Transformer-XL) and its offline model is larger than that between LC-BLSTM and BLSTM. The offline Transformer outperforms BLSTM by a considerable margin, while Transformer-XL is only comparable with LC-BLSTM in terms of WERs. Though the Transformer with attention mask as [−∞,40] can outperform LC-BLSTM, it is not computationally feasible during inference. This observation may align well with the argument that Transformers are more powerful to capture the long-term correlations in sequential signals. In a scenario with limited feature context, however, Transformers are hindered to release their modeling power.