Adversarial Training for Multilingual Acoustic Modeling
Table 1: Comparison of the WERs across monolingual, multilingual and multilingual adversarial models for each language. The average relative WER reductions w.r.t. monolingual models are shown in parentheses.
['Language', 'Training Data (hr)', 'WER (%) of different models Monolingual', 'WER (%) of different models Multilingual', 'WER (%) of different models Adversarial']
We train the models based on seven languages including English, Spanish, French, Portuguese, Italian, Russian and Dutch. These languages come from different groups including Romance, Germanic, and Slavic languages. The training data are manually segmented and transcribed yielding high-quality transcripts for training. To obtain alignments for CE training, the data are force-aligned by using acoustic model trained by supervised training data (ranging from 71 to 363 hours) for each language. The test data contain at least 5 hours of data for each language. Note that for Dutch, we purposely use a small amount of training data, i.e. 13.5 hr, to simulate a low-resource condition. We want to analyze how such a low-resource language can benefit from multilingual and adversarial training. The inputs to the models are 80-dimensional log-mel filterbank energy features based on a 25-ms frames width a 10-ms frame rate. For each language, a new state inventory consisting of 1600 CD states is generated. For monolingual models, we note that the relative high WERs for Russian and Dutch are due to their insufficient training data. Multilingual training improves WER for almost all languages except French. The improvement for Dutch is the most, i.e. a relative 17%. The degradation for French indicates that multilingual data introduces noise for its feature representation. However, we will show that adversarial training improves in this condition. Overall, the multilingual model performs better than monolingual models by 6% in terms of WER. The WER reduction ranges from 1.5% to 9%. The improvement for Dutch is reduced compared to the monolingual-to-multilingual gain since it has obtained significant gain from multilingual training. The gains brought by adversarial training come from enforcing the shared layers to learn language-invariant features, and it seems that most languages benefit from this enhanced knowledge sharing. The degradation of English may be due to that it already gained large improvement from monolingual to multilingual training, and learning language-invariant features reduced the benefits. In summary, the multilingual adversarial model performs uniformly and significantly better than monolingual models for all languages, and the average WER reduction is around 10%. We will further analyze the outputs from the shared layers to confirm whether language specific information is attenuated in Sect. On average, adversarial training brings an additional relative 4% WER reduction over multilingual training.