Learning to Recognise Words using Visually Grounded Speech
Table 1: Speech caption-to-image retrieval scores including 95% confidence intervals for our model. For comparison, the models of Merkx et al. [merkx2019language], Chrupała et al. [chrupala2017representations] and Harwath et al. [harwath2015deep] which were also trained on Flickr8k speech captions are provided.
['[BOLD] Model', 'R@1', 'R@5', 'R@10', 'Med. R']
This indicates how well the model learned to embed the speech and images in the common embedding space. R@N is the percentage of items for which the correct image was in the top N retrievals. Median R is the median rank of the correctly retrieved image. The addition of an extra GRU layer has led to a substantial performance increase, allowing dependencies in longer speech captions to be captured better.