First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs
Table 1: Word error rate (WER) and character error rate (CER) results from a BDRNN trained with the CTC loss function. As a baseline (No LM) we decode by choosing the most likely label at each timestep and performing standard collapsing as done in CTC training. We compare this baseline against our modified prefix-search decoder using a dictionary constraint and bigram language model.
['Model', 'CER', 'WER']
We trained a BRDNN with 5 hidden layers, all with 1824 hidden units, for a total of 20.9M free parameters. The third hidden layer of the network has recurrent connections. We use the Nesterov accelerated gradient optimization algorithm as described in Sutskever et al. After each full pass through the training set we divide the learning rate by 1.2 to ensure the overall learning rate decreases over time. We train the network for a total of 20 passes over the training set, which takes about 96 hours using our Python GPU implementation. For decoding with prefix search we use a beam size of 200 and cross-validate with a held-out set to find a good setting of the parameters α and β.