Massively Multilingual Adversarial Speech Recognition
Table 2: Word error rate (%) comparison of multilingual models adapted to target languages, with and without auxiliary training objectives (relative change in parentheses). Additionally including Cyrillic-script languages in pretraining (Cyr) doesn’t consistently improve over a model pretrained on Quechuan languages (Que) unless additional phoneme and language-adversarial objectives (+phn and +adv) are used in combination (+phn+adv). The auxiliary objectives help when pretraining languages are varied, but hinder when they are very similar. The final four columns suggest that the objectives are complementary. Average relative word error rate change for each pretraining set when adding in the auxiliary objectives (versus no aditional objectives) is indicated by Avg. rel. \Delta.
['Target', 'Mono', 'Que -', 'Que +phn+adv', 'Que +phn+adv', 'Cyr -', 'Cyr +phn+adv', 'Cyr +phn+adv', 'Que+Cyr -', 'Que+Cyr +phn', 'Que+Cyr +adv', 'Que+Cyr +phn+adv', 'Que+Cyr +phn+adv']
Cyr pretraining underperforms pretraining with Que for all evaluation languages likely due to the orthographic mismatch with all of the evaluation languages. The model pretrained on Que+Cyr also underperforms Que. Introducing the auxiliary phoneme and language-adversarial objectives helps to overcome this performance loss, making the Que+Cyr-pretrained model the best for adaptation to Aymara and Indonesian. Que remained the best pretraining set for adaptation to SB Quechua, which is unsurprising given how well represented SB Quechua is by the languages included in the Quechuan language group. This suggests that when a substantial amount of data in very closely related languages is available (in this case, close to 100 hours of Que data), then there is little to be gained from highly unrelated languages.