CSTNet: Contrastive Speech Translation Network for Self-Supervised Speech Representation Learning
Table 3: ABX Scores on ZRC19 Challenge English Test Set
['[BOLD] Method', '[BOLD] ABX']
We compute the ABX score using all the layers of the pre-trained CSTNet’s audio encoder trained on different language pairs. Here, we present the best results that is usually obtained using the representations in the middle of the network (for layer numbers 5, 6). The network hits a sweet spot in the middle layers, where the receptive field is approximately 100-140 ms. A similar trend is observed for all the languages. We significantly outperform Wavenet-VQ (ZS), the best performing submission to the ZRC19 challenge, based on Vector Quantization VAE (VQ-VAE) To have a fair comparison, we also compare our model against Wavenet-VQ (MuST-C) that is trained on the English speech from the MuST-C corpus on which CSTNet is also trained. CSTNet still outperforms Wavenet-VQ. Hence, we show that our framework could be an alternative to reconstruction based representation learning methods.