Toward Computation and Memory Efficient Neural Network Acoustic Models with Binary Weights and Activations
Table 4: WERs (%) of neural network with binary weights and activations on the AMI dataset. The number of hidden units is 2048, and b denotes binarization.
['Model', '[ITALIC] b', 'dev', 'eval']
For the binary weight systems, we revisited the semi-stochastic binarization approach. While the convergence curves were similar Since the model was initialized from a Sigmoid network, the binary activation system with (0,1) worked much better than its counterpart with (−1,+1) when k=1. Again, we did experiments by tuning the threshold k for binary activation systems. Unlike the experiments on WSJ1, the models are relatively tolerant to changing k with (−1,+1) for binarization, and we only observed divergence when k=7. This may be because that we used different features in these experiments, causing differences in the distributions of ^hl. However, this is not the case for binarization with (0,1), as the system degrades rapidly when k increases. The reason may be that the binarization function with (0,1) is not symmetric, and the approximation error using an identity function is significant for large k when inputs are negative. Again, we failed to train binary neural networks with 2048 hidden units due to divergence in training.