AVLnet: Learning Audio-Visual Language Representations from Instructional Videos
(a) YouCook2
['[BOLD] Method', '[BOLD] Training Set', '[BOLD] Video Clip Retrieval - YouCook2 Mod.', '[BOLD] Video Clip Retrieval - YouCook2 R@1', '[BOLD] Video Clip Retrieval - YouCook2 R@5', '[BOLD] Video Clip Retrieval - YouCook2 R@10', '[BOLD] Video Clip Retrieval - YouCook2 Md. R', '[BOLD] Language Retrieval - YouCook2 Mod.', '[BOLD] Language Retrieval - YouCook2 R@1', '[BOLD] Language Retrieval - YouCook2 R@5', '[BOLD] Language Retrieval - YouCook2 R@10', '[BOLD] Language Retrieval - YouCook2 Md. R']
AVLnet outperforms all prior models on YouCook2 (zero-shot and fine-tune), with a 11.8% and 27.7% absolute increase in performance at R@10 over the previous state-of-the-art for the video clip and language retrieval tasks respectively. Overall, our results indicate that AVLnet has learned powerful language representations despite never seeing text and only using raw audio.