Learning Word Embeddings from Speech
Table 1: The Spearman’s rank correlation coefficient ρ between the rankings produced by each model against the human rankings. #(word pairs) is the number of word pairs in the dataset, and #(not found) is the number of word pairs whose vector representations could not be found.
['No.', 'Dataset', '#(word pairs)', 'Seq2seq Audio2vec #(not found)', 'Seq2seq Audio2vec  [ITALIC] ρ', 'GloVe Wikipedia 2014 #(not found)', 'GloVe Wikipedia 2014  [ITALIC] ρ']
Verb-143 These 13 benchmarks contain different numbers of pairs of English words that have been assigned similarity ratings by humans, and each of them tries to evaluate the word vectors in terms of different aspects. For example, RG-65 and MC-30 focus on nouns, YC-130 and SimVerb-3500 focus on verbs, and Rare-Word focuses on rare-words. We compared the vector representations learned by Seq2seq Audio2vec with GloVe trained on Wikipedia 2014. The similarity between a given pair of words was calculated by computing the cosine similarity between their corresponding vector representations. This demonstrates that our proposed Seq2seq Audio2vec is capable of capturing semantic information from raw speech and representing it in a fixed-length vector representation, although the scores of our model were consistently lower than that obtained by GloVe. Aside from the differences due to the speech and text training data, we believe the reason for this difference is due to the inherent variability in speech production. Unlike textual representations, every instance of any spoken word ever uttered is different, due to vocal tract differences across speakers, speaking styles, contextual differences, and environmental conditions, to name but a few of the major influences on a speech recording. Clearly, one of the challenges for learning semantics directly from raw speech is to derive a more robust mechanism to address these issues. To us, what is more impressive is that many of the test scores are close.