Multi-view Frequency LSTM: An Efficient Frontend for Automatic Speech Recognition
Table 1: Various FLSTM and mvFLSTM topologies and their corresponding network size expressed in total number of trainable parameters (Pars.) compared to a baseline time LSTM acoustic model with 5 layers and hidden size of 768. The mvFLSTM topologies contain up to 3 FLSTMs with specified number of layers and layer size (e.g. an FLSTM with 2 FLSTM layers of size 16 hidden cells is denoted as L2x16), and different window size and stride (e.g. 48/24 denotes a window size of 48 frequency bins with a stride of 24 bins). The mvFLSTM models that have an additional projection layer are named mvFLSTMp.
['id', 'model', 'FLSTM layers', 'FLSTM1', 'FLSTM2', 'FLSTM3', 'Proj. dimension', 'Total Pars. (M)', 'Δ Pars. (%)']
All FLSTM models feed into a time LSTM with the same architecture as that of the baseline AM. The second column denotes the number and hidden size of the FLSTM layers that were used for each stack of FLSTM layers. For the purpose of this paper, each FLSTM stack had the same number of layers and layer size. For the mvFLSTM models, columns 3 to 5 present the window size and stride used in each of the FLSTM components and are denoted as Fi/Si. Column 5 indicates whether a projection layer was used and indicates the corresponding dimension after dimensionality reduction. The last two columns respectively show the total amount of trainable parameters for each configuration and its relative increase compared to the baseline LSTM AM. From the table, it is clear that the projection layer in the multi-view FLSTM configurations limits the increase in parameters as discussed above. Models with 3 views and projected dimension up to 512, have similar amount of free parameters as the single-view FLSTM model with window size/stride of 48/24, despite the use of wider and deeper stacks of FLSTM layers (column 2).