Toward Computation and Memory Efficient Neural Network Acoustic Models with Binary Weights and Activations
Table 3: WERs (%) of neural networks with binary weights and activations on the WSJ1 dataset. We set p=0 for the system with binary weights, and k=1 for the system with binary activations.
['Model', 'Size', '[ITALIC] b', 'dev93', 'eval92']
For all the experiments in this table, the weights in the input and Softmax layer were fixed, and the first and last hidden layers used Sigmoid activations. Using a larger number of hidden units works slightly better for both binary weight and binary activation systems. However, we only managed to train the network with 1024 hidden units and achieved much inferior accuracies. Training fully binary neural networks is still a challenge from our study. With Sigmoid activations, using (0, 1) for binary weights can cause training divergence as the elements of the activation vector ^hl are always positive. Using (0,1) for binary activations, however, we achieved lower WER. The reason may be that the network was initialized from Sigmoid activations, and (0,1) is much closer to Sigmoid compared to (−1,1). In fact, (0,1) binary function can be viewed as the hard version of Sigmoid. Using (−1,+1) for binary activations may work better with networks initialized from Tanh activations, and that will be investigated in our future works.