The IBM 2015 English Conversational Telephone Speech Recognition System
Table 1: Word error rates of sigmoid vs. Maxout networks trained with annealed dropout (Maxout-AD) for ST CNNs, DNNs and score fusion on Hub5’00 SWB. Note that all networks are trained only on the SWB-1 data (262 hours).
['Model', 'WER SWB (ST) sigmoid', 'WER SWB (ST) Maxout-AD']
All Maxout networks utilize 2 filters per hidden unit, and the same number of layers and roughly the same number of parameters per layer as the sigmoid-based DNN/CNN counterparts. Parameter equalization is achieved by having a factor of √2 more neurons per hidden layer for the maxout nets since the maxout operation reduces the number of outputs by a factor of 2. Note that ReLU networks, in our experience, perform on-par with sigmoid-based DNNs in this data regime. Maxout networks trained with AD (Maxout-AD), on the other hand, show a clear advantage over our traditional networks. Also, note that the convolutional layers of the Maxout-AD CNN have only 128 and 256 feature map outputs, whereas those of the sigmoid CNN has 512/512 outputs. Training of the Maxout-AD CNN with a 512/512 filter configuration is in progress.