From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning
Table 1: Image retrieval accuracy at 5 and at 10 on test data for the versions of Word Sum, Word GRU and Phon GRU chosen by validation.
['Model', 'Acc @ 5', 'Acc @ 10']
The scores for the Word GRU are comparable to what \newcitechrupala2015learning report for their multitask Imaginet model, whose visual pathway has the same structure, and who use the same data. More recently, \newcitevendrov2015order report substantially higher scores for image search with a word-level GRU model, with the following main differences from our setting: better image features, larger training set, and a loss function optimized for the ranking task. The proportion of phonemes preceding a word boundary is 0.29, meaning that predicting no word boundary by default would be correct in 0.71 of cases. At the highest hidden layer, enough information about the word form is available for correct prediction in 0.82 of cases – substantially above the majority baseline. The lower levels allow for more accurate prediction of word boundaries: 0.86 at the middle hidden layer, and 0.88 at the bottom level. Prediction scores of the logistic regression model based on the activation patterns of the lowest hidden layer are comparable to those of a bigram logistic regression model.