Toward Computation and Memory Efficient Neural Network Acoustic Models with Binary Weights and Activations
Table 1: WERs (%) of binary weight networks on WSJ1. The number of hidden units is 1024 for experiments in this table.
['Model', 'Input', 'Softmax', 'dev93', 'eval92']
In our experiments, training neural networks with binary weights from random initialization usually did not converge, or converged to very poor models. We addressed this problem by initializing our models from a well-trained neural network model with real-valued weights. This approach worked well, and was used in all our experiments. We trained the baseline neural network with an initial learning rate of 0.008, following the nnet1 recipe in Kaldi. This initial learning rate was found to be a good tradeoff between convergence speed and model accuracy. Here, we explored several settings: the weights in the input layer and Softmax layer were binary, real-valued or fixed from initialization. We also did experiment to update those real-valued weights jointly with the binary weights using the same learning rate, but obtained much worse results. The reason may be that the gradients of the real-valued weights and binary weights are in very different ranges, and updating them using the same learning rate is not appropriate. In order to have a complete picture, we have also tried using binary weights for the input layer and the Softmax layer. As expected, we achieved much lower accuracy, confirming that reducing the resolution of the input features and activations for the Softmax classifier are harmful for classification. Applying this step very frequently is harmful to the SGD optimization as it can counteract the SGD update. In our experiments, we set a probability p to control the frequency of this operation. More precisely, after each SGD update, we draw a sample from a uniform distribution between 0 and 1, and if its value is smaller than p, then the semi-stochastic binarization approach will be applied. Therefore, a larger p means more frequent operations and vice versa. Note that, we used the dev93 set to choose the language model score for the eval92 set, and from our observations, the results of development and evaluation sets are usually not well aligned, demonstrating that there may be a slight mismatch between the two evaluation sets. The semi-stochastic binarization approach will be revisited on the AMI dataset.