Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces
Table 3: Accuracy on spoken word classification. ENls−enswc means that the speech and text embeddings were learned from the speech training data of English LibriSpeech and text training data of English SWC, respectively, and the testing audio segments came from English LibriSpeech. The same rule applies to Table 5 and Table 6. For the Word Classifier, ENls−enswc and ENswc−enls could not be obtained since it requires parallel audio-text data for training.
['Corpora', 'ENls−enls', 'FRls−frls', 'ENswc−enswc', 'DEswc−deswc', 'ENls−enswc', 'ENswc−enls']
We observe that the accuracy decreases as the level of supervision decreases, as expected. We also note that although the Word Classifier significantly outperforms all the other approaches under all corpora settings, the prerequisite for training such a fully-supervised approach is unrealistic—it requires the utterances to be perfectly segmented into audio segments corresponding to words with the word identity of each segment known. We emphasize that the Word Classifier is just used to establish an upper bound performance that gives us an idea on how good the classification results could be. The aforementioned analysis applies to different corpora settings. We think this is because the embedding spaces learned from the same corpora (e.g., both embeddings were learned from LibriSpeech) exhibit higher similarity than those learned from different corpora, making the alignment more accurate.