Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks
Table 1: ABX (%) on English for different discrete latent-variable neural models with and without speaker conditioning.
['Model', 'ABX on decoder output No spkr cond.', 'ABX on decoder output Spkr cond.', 'ABX on latent', 'Bitrate']
K is set so that 512 unique symbols can be obtained in all models, but not all the symbols are necessarily utilised, so the bitrates differ. The form of the latent symbols z is very different for the different methods: binarised for STE, one-hot for CatVAE, or an embedding selected from a codebook for VQ-VAE. The cosine-DTW distance metric used in ABX might not be equally well-suited to all of these. To make a fairer comparison, we pass the encoded symbols through the decoder for each model, and compare all the models on the reconstructed filterbank ^y obtained at the output from each AE. All methods are therefore compared based on the same feature type. In all cases, VQ-VAE performs best. We also observe that the CatVAE’s one-hot latent symbols are particularly ill-suited to direct ABX evaluation: 45.6% error rate compared to 24.3% and 28.7% when using decoder outputs. Our main hypothesis is that decoupling unit discovery from speaker modelling produces intrinsically better discrete representations. Our best overall result of 22.1% when using the VQ-VAE is better than both of these. This is particularly noteworthy since this model outperforms both the input and output features on which it is trained, and it does so while performing intermediate discretisation which encodes the input at a much lower bitrate. Discretisation on its own (ABX of 26.0% for VQ-VAE) already leads to improvements over filterbanks (29.5%), but it is the combination of discretisation and decoupled speaker conditioning which leads to the best score (22.1%).