Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training
Table 1: Evaluation on pre-training tasks and datasets using TVR, TVQA, Howto100M-R and Howto100M-QA validation set as benchmarks. Dark and light grey colors highlight the top and second best results across all the tasks trained with TV Dataset. The best results are in bold. For simplicity, we only report video moment retrieval results for TVR and Howto100M-R.
['Pre-training Data', '[EMPTY]', 'Pre-training Tasks', 'TVR R@1', 'TVR R@10', 'TVR R@100', 'TVQA Acc.', 'Howto100M-R R@1', 'Howto100M-R R@10', 'Howto100M-R R@100', 'Howto100M-QA Acc.']
To evaluate the effectiveness of each pre-training task, we conduct ablation experiments through pre-training on TV dataset only. When MLM, MNCE and FOM are jointly trained (L3), there is a large performance gain in accuracy on TVQA and significant improvement on the two Howto100M downstream tasks. Comparable results are achieved on TVR. This indicates that FOM, which models sequential characteristics of video frames, can effectively benefit downstream tasks that rely on temporal reasoning (such as QA tasks).