Audio-attention discriminative language model for ASR rescoring
Table 4: Model size and Relative Word Error Rate Reduction (WERR) for different attention location
['[BOLD] Encoder type', '[BOLD] # params (MM)', '[BOLD] WERR']
In addition to attending to hidden layer outputs of the first pass LSTM-AM, we also experimented with applying a learnable context encoders on top of the LSTM AM outputs. Following various papers, we experimented with three different encoder types: a) 2-layer pyramidal LSTM (PyLSTM) We see that using Pyramidal LSTM and TDNNs for encoding the input improves the performance of the model. However, both TDNN and Pyramidal LSTM have a significant increase in number of parameters as well as training and inference time. CNN encoders are able to improve the performance over having no encoder. For all our other experiments, we chose to go with CNN encoder because of its fast training and inference speed.