Phoneme-Based Contextualization for Cross-Lingual Speech Recognition in End-to-End Models
Table 1: Comparison of WERs (%) of different models for a foreign place name recognition task and Google general English traffic.
['Model', 'Biasing unit', '[ITALIC] Directions w/o Bias', '[ITALIC] Directions w/ Bias', 'General Traffic (English)']
We compare the biasing results of the wordpiece-phoneme model to a grapheme-only model and a wordpiece-only model. The latter two models have the same structure as the wordpiece-phoneme model. The difference is that the grapheme model has 76 graphemes as outputs and the wordpiece model has 4,096 wordpieces. This leads to around 117M and 120M parameters for the grapheme model and wordpiece model, respectively. For these two models, biasing is done at the grapheme level or wordpiece level alone \addedusing the English transliterated versions of French biasing words. First, we see that all three models perform poorly without biasing. This is because the place names are in French and they have never been seen in training, The WER reductions range from 9%-23% relatively for different models when compared to the no-bias case. Comparing different biasing strategies, we find that the wordpiece-phoneme model performs the best: 16% relatively better than the grapheme model, and 8.3% better than the wordpiece model. Since the wordpiece-phoneme model contains both wordpieces and phonemes as modeling units, we can further perform wordpiece biasing in addition to phoneme-based biasing by building a wordpiece FST in parallel to the phoneme FST. This shows that wordpiece and phoneme biasing are complementary to each other. We note that the same weights are used for both phoneme and wordpiece biasing, and empirically we did not find significant improvements by using different weights. To ensure there is no regression in no-biasing scenarios, we compare three models in decoding regular English utterances from general Google traffic. In decoding, we turn the biasing mechanism off by using an empty list of biasing phrases. The wordpiece-phoneme model performs a little better than the grapheme model, and we attribute that to \replacedthe higher frequency of wordpieces during trainingdominating wordpieces in the model. Compared to the wordpiece model, the wordpiece-phoneme model has a slight degradation (0.1% absolute WER). This is due to the introduction of phonemes in modeling.