Towards Structured Deep Neural Network for Automatic Speech Recognition
Table 1: Phoneme Error Rate(%). Rows (1) and (2) are for two different acoustic vector sequence inputs. Column (A) are the results of structured SVM, while columns (B) and (C) are for structured DNN respectively with the lost function to approximate phoneme accuracy(Approx. Ph. Acc) in Subsection 2.3.1 and maximizing margin(max. Margin) in Subsection 2.3.2. Column (D) is for the extension of column (C) into the Full-scale structued DNN (FSDNN) described in Subsection 2.6. Column (E) is the baseline (Vesel’s recipe).
['Phoneme Error Rate(%)', '(A) structured SVM', '(B) structured DNN', '(C) structured DNN', '(D) FSDNN', '(E) baseline']
Rows (1) and (2) are for different acoustic features, where row (1) is actually the acoustic features used by Vesel’s recipe. Column (A) are the results of structured SVM. Column (E) is the baseline. Margin performed much better than approximating the phoneme accuracy (columns (C) v.s. (B)). A possible reason is as follows. The loss function of approximating the phoneme accuracy may be inevitably dominated by negative examples, since we used much more negative examples than positive examples in training, and the positive and negative examples were equally weighted. The loss function of maximizing the margin, on the other hand, focused on the score difference between each pair of positive and negative examples, and as a result, the positive and negative examples were weighted equally even if very different numbers. The next experiment is to analyze the PER for different choices of the key hyper-parameters for the full-scale structured DNN (FSDNN), number of hidden layers L and number of neurons M in each hidden layer. The horizontal axis is M where M=100,200,...1000, and the vertical axis is L where L=1,2,...5. Therefore, the figure consists of 5×10=50 data points. The overall performance is approximately between 17% and 19%, more or less comparable to baseline. For this task, N=1 for training, N=10 for inferencing. Better PER seemed to be located at several disjoint regions (4 valleys in the figure). The disjoint valleys may come from the relatively poor learning due to lack of training data and poor initialization. like, which may result in a similar behavior of ReLU training (highly dependent on the initialization).