Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training
Table 2: Comparison between a flat BERT-like encoder (F-Trm), a Hierarchical Transfomrer (H-Trm) baseline and Hero using TVR and TVQA validation set as benchmarks. Results in the last two rows are obtained from pre-training the models with MLM + MNCE + FOM + VSM on TV Dataset. For simplicity, we report only video moment retrieval for TVR.
['Pre-training', 'Model', 'R@1', 'TVR R@10', 'R@100', 'TVQA Acc.']
(i) When no pre-training is applied, F-Trm is much worse than Hero on both tasks. H-Trm achieves comparable results to Hero on TVR, but worse on TVQA. Unlike F-Trm, H-Trm and Hero explicitly utilize the inherent temporal alignment between two modalities of videos, which is uniquely important for video+language tasks. (ii) With pre-training, Hero shows significant improvement over H-Trm. Our hypothesis is that with the hierarchical design, Hero can capture cross-modal interactions between visual frames and its local textual context better than H-Trm. Such cross-modality joint understanding of visual and textual contexts is critical for video-based retrieval and QA tasks. (iii) Pre-training lifts Hero performance by a large margin, but not very helpful for H-Trm. These results provide strong evidence that cross-modal interactions and temporal alignments between visual frames and its local textual context learned by Hero are essential for these video+language tasks.