Transcription-Enriched Joint Embeddingsfor Spoken Descriptions of Images and Videos
Table 1: In this table we present the recall scores in both datasets and with or without adding text at train time. The difference between the 3rd and 4th line (marked with *) is that the 4th are the results published at [5] while the 3th are our results of the same experiment. We used the implementation they provided, but couldn’t have the same experimental setup, reducing the batch size from 128 to 80. Results are on test dataset for EPIC and validation for Places
['Text', 'Dataset', 'Img R@1', 'Img R@5', 'Img R@10', 'Audio R@1', 'Audio R@5', 'Audio R@10']
We think this is due to two main reasons. Firstly, textual transcriptions can be understood as a clean version on audio data, and it encodes in a much cleaner way the concept in the video clip. For this reason, it is no surprise that it helps as an intermediate modality between audio and image. Secondly, pretrained BERT is the state-of-the-art in text embeddings. This model had already discovered concepts during its pretraining. For this reason, we do not need the audio and image branches (trained from scratch) to discover the concepts on their own, and find BERT as almost a ground truth embedding they need to learn the mapping to.