Language learning using Speech to Image retrieval
Table 1: Image-Caption retrieval results on the Flickr8k test set. R@N is the percentage of items for which the correct image or caption was retrieved in the top N (higher is better). Med r is the median rank of the correct image or caption (lower is better). We also report the 95 percent confidence interval for the R@N scores.
['Model', 'Caption to Image R@1', 'Caption to Image R@5', 'Caption to Image R@10', 'Caption to Image med r']
The caption embeddings are ranked by cosine distance to the image and vice versa where R@N is the percentage of test items for which the correct image or caption was in the top N results. We compare our models to [Harwath2015] and [Chrupala2017], and include our own character-based model for comparison. [Harwath2015] is a convolutional approach, whereas [Chrupala2017] is an approach using recurrent highway networks with scalar attention. The character-based model is similar to the model we use here and was trained on the original Flickr8k text captions (see [merkx2019] for a full description). Both our MFCC and MBN based model significantly outperform previous spoken caption-to-image methods on the Flickr8k dataset. The largest improvement is the MBN model which outperforms the results reported in [Chrupala2017] by as much as 23.2 percentage points on R@10. The MFCC model also improves on previous results but scores significantly lower than the MBN model across the board, improving as much as 12.3 percentage points over previous work. There is a large performance gap between the text-caption to image retrieval results and the spoken-caption to image results, showing there is still a lot of room for improvement.