Visually grounded cross-lingual keyword spotting in speech
Table 2: Cross-lingual keyword spotting results (%) on test data.
['Model', '[ITALIC] P@10', '[ITALIC] P@ [ITALIC] N', 'EER', 'AP']
Without seeing any speech transcriptions or translated text, XVisionSpeechCNN achieves a P@10 of 58%, with XBoWCNN the only model to outperform the visually grounded model. By comparing performance to DETextPrior, we see that XVisionSpeechCNN is not just predicting common German words. Interestingly, XVisionSpeechCNN also outperforms DEVisionCNN over all metrics. If the former were perfectly predicting the German visual tags (which is what it is trained to do), then the performance of these two models would be the same. We see, however, that XVisionSpeechCNN is doing more than simply mapping the acoustics to the visual tags; we speculate that it is therefore picking up information in the speech which cannot be obtained from the corresponding test images.