UWSpeech: Speech to Speech Translation for Unwritten Languages
Table 1: The BLEU scores of Spanish to English speech to speech translation, where English is taken as the unwritten language.
['Method', '[ITALIC] Direct Translation', '[ITALIC] VQ-VAE', '[ITALIC] UWSpeech']
VQ-VAE achieves slightly better BLEU score than Direct Translation, but still with poor accuracy, which demonstrates the limitations of the purely unsupervised method for speech discretization when handling speech translation. UWSpeech achieves 17.33 BLEU points, about 10 points higher than VQ-VAE and 16 points higher than Direct Translation. We also find that the inverter in XL-VAE can get a lower reconstruction loss than VQ-VAE on the validation set, demonstrating that the discrete tokens extracted by XL-VAE can not only help the discrete token translation in translator but can also benefit the speech reconstruction in inverter, which together contribute to the better accuracy in speech translation. The above results demonstrate the advantages of XL-VAE in leveraging cross-lingual speech recognition for speech discretization and the effectiveness of UWSpeech for unwritten speech translation.