Training Multi-Speaker Neural Text-to-Speech Systems using Speaker-Imbalanced Speech Corpora
Table 1: Data sets of target speakers.
['[BOLD] Speaker ID', '[BOLD] XS01', '[BOLD] XS02', '[BOLD] S03', '[BOLD] S04', '[BOLD] S05', '[BOLD] M06', '[BOLD] M07', '[BOLD] M08', '[BOLD] L09', '[BOLD] XL10']
Our experiments are data-driven and we seek to identify the best approach to train a speech synthesis system from an imbalanced speech corpus. The corpus we used contained utterances from ten female Japanese speakers, who are professional or at least familiar with voice acting work. The number of utterances of each speaker ranged from 1,000 to 10,000. The main investigation in this paper is which methodology efficiently uses an imbalanced multi-speaker corpus to improve performance for the generated speech of all speakers involved. Multiple strategies are compared in the experiments: SD: This is our baseline strategy. UN: A multi-speaker model trained with an under-sampled corpus containing 753×10 utterances. Each speaker contributes 735 utterances to this corpus, where 735 is the number of utterances from speaker XS01, who has the least amount of training data. MU: The conventional multi-speaker models trained with all the data from every speaker, i.e., all 32,076 training utterances from the original corpus. OV: A multi-speaker model trained with an over-sampled corpus. We used all utterances and then sampled more from minority speakers so that each got the same frequency in training. The amount of training data is 8,750×10 utterances. E1, E2, E3: Multi-speaker models trained with resampled corpora. In total, 3,000 utterances are sampled with replication from each speaker. EN: A non-parametric ensemble model. The sampling strategies E1, E2, and E3 seem to be better than the baseline SD but worse than MU. The performances vary for each session due to the stochastic nature of the sampling method. In general, the ensemble strategy EN had the best results.