Learning to Speak Fluently in a Foreign Language:Multilingual Speech Synthesis and Cross-Language Voice Cloning
Table 1: Speaker similarity Mean Opinion Score (MOS) comparing ground truth audio from speakers of different languages. Raters are native speakers of the target language.
['Source Language', 'Target Language EN', 'Target Language ES', 'Target Language CN']
For each language, we chose one speaker to use for similarity tests. The CN speaker has more natural variability compared to EN and ES, leading to a lower self similarity. The scores are consistent when EN and CN raters evaluate the same EN and CN test set. However, when rating synthetic speech, we observed that English speaking raters often considered “heavy accented” synthetic CN speech to sound more similar to the target EN speaker, compared to more fluent speech from the same speaker. This indicates that accent and speaker identity are not fully disentangled. We encourage readers to listen to samples on the companion webpage. This is a consequence of the low speaker coverage of CN compared to EN in the training data, as well as the large distance between CN and other languages.