Visually grounded cross-lingual keyword spotting in speech
Table 4: Cross-lingual keyword spotting results (%) for different variants of XVisionSpeechCNN on development data.
['Model', '[ITALIC] P@10', '[ITALIC] P@ [ITALIC] N', 'EER', 'AP']
Variants and ideal supervision. We compare different variants of XVisionSpeechCNN to gain insight into properties of the model. XVisionSpeechCNN produces scores →f(X)∈[0,1]W for all W=1k words in its output vocabulary. But we are actually only interested in those dimensions w corresponding to the test keywords. If we knew the keywords at training time, we could train a model which only tries to predict the visual tags corresponding to these keywords. Performance is similar to that of XVisionSpeechCNN, with the latter being slightly better on most metrics. This effectively regularises our model (improving results). XVisionSpeechCNN is trained on soft scores from a visual tagger. What if we had the true hard assignments from the manual annotations for the training images? OracleXVisionSpeechCNN is trained on such oracle targets. They described this as a student-teacher approach, where the student (in our case the speech network) is trying to distil knowledge from the teacher network (in our case the visual tagger).