Frame Stacking and Retaining for Recurrent Neural Network Acoustic Model
Table 2: the CERs of different FR with the same FS for a 4-layers LSTM model.
['[ITALIC] FS', '[ITALIC] FR', '[BOLD] CER']
Frame stacking cuts down the number of input frames, so it leads to almost linear speedup of training. But when its model is applied in the decoder directly, it will cause the great CER degradation, as the modeling duration does not match. The decoding network is generated to fit for original modeling duration. N frames corresponds only one input feature vectors of decoder for frame stacking, while N frames corresponds N of them for original modeling. Therefore, frame retaining in decoder could match the number of input feature vectors and that of frames. We denote the number of non-overlapping stacked frames as FS, and the times of a super frame retaining as FR. the other modeling duration mismatch also results in worse performance of decoder. FR being 1 means no frame retaining, and it demonstrates that only frame stacking could not improve the accuracy of non-CTC neural network.