Sound-Word2Vec: Learning Word Representations Grounded in Sounds
Table 1: Text-based sound retrieval (higher is better). We find that our sound-word2vec model outperforms all baselines.
['Embedding', 'Recall @1', 'Recall @10', 'Recall @50', 'Recall @100']
Given a textual description of a sound as query, we compare it with tags associated with sounds in the database to retrieve the sound with the closest matching tags. Note that this is a purely textual task, albeit one that needs awareness of sound. In a sense, this task exactly captures what we want our model to be able to do – bridge the semantic gap between language and sound. We use the training split (Sec. For retrieval, we represent sounds by averaging the learnt embeddings for the associated tags. We embed the caption provided for the sound (in the Freesound database) in a similar manner, and use it as the query. We then rank sounds based on the cosine similarity between the tag and query representations for retrieval. We evaluate using standard retrieval metrics – Recall@{1,10,50,100}. Note that the entire testing set (≈10k sounds) is present in the retrieval pool. So, recall@100 corresponds to obtaining the correct result in the top 1% of the search results, which is a relatively stringent evaluation criterion. Results. Table. Among our approaches, tag-word2vec performs second best – this is intuitive since the tag distributions implicitly capture auditory relatedness (a sound may have tags cat and meow), while word2vec and sound-word2vec(r) have the lowest performance.