AVLnet: Learning Audio-Visual Language Representations from Instructional Videos
Table 2: Analysis of AVLnet and AVLnet-Text design choices. Performance of each compared model is reported as video clip retrieval R@10.
['[BOLD] Experiment', '[BOLD] Configuration', '[BOLD] YouCook2  [BOLD] Zero-Shot', '[BOLD] YouCook2  [BOLD] Fine-Tune', '[BOLD] MSR-VTT  [BOLD] Zero-Shot', '[BOLD] MSR-VTT  [BOLD] Fine-Tune', '[BOLD] LSMDC  [BOLD] Zero-Shot', '[BOLD] LSMDC  [BOLD] Fine-Tune']
For AVLnet, we investigate the effect of adding text to the audio-visual model during evaluation and fine-tuning, changing the HowTo100M clip sampling method, and varying the loss function. For AVLnet-Text, we investigate the effect of using independent audio and text branches as compared to our language fusion technique. We report each model’s video clip retrieval R@10 on all three evaluative datasets in the zero-shot setting and after fine-tuning. Adding text from the downstream datasets. We evaluate the performance of AVLnet trained on audio and video from HowTo100M and fine-tuned/evaluated on audio, video, and text from YouCook2, MSR-VTT, and LSMDC. This experiment represents the scenario where obtaining text annotations during training is expensive, but text exists or can be obtained for smaller evaluative datasets or real world applications. M is beneficial. Further, this result suggests that AVLnet learns language representations from speech, not just natural sounds or voice characteristics. HowTo100M clip selection. that used video clips segmented at the ASR speech boundaries. Loss functions. We find the MMS loss and the NCE loss to outperform the Max-Margin Ranking loss, prompting us to use the MMS loss in our experiments. Processing text with an independent branch. The MMS loss is applied over each of the modality pairs (audio-video, audio-text, and video-text), and the branches are jointly optimized through the sum of these three losses. During evaluation, we use the sum of the audio and text embedding vectors to retrieve video clips.