Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA
Table 3: Model Ablation: our dual-attention / frame-selection Gates, new loss functions, and dense captions help improve the model’s performance (NewLoss: IOFSM+BBCE).
['[EMPTY]', 'Model', 'Val Score (%)']
Each of our dual-attention and frame selection gates alone shows a small improvement in performance than the baseline (row 3 vs 1 and 2 vs 1, respectively). However, when they are applied together, the model works much better. The reason why they are more effective when put together is that frame selection gates basically select frames based on useful information from each frame feature and our dual-attention can help this selection by getting more relevant information to each frame through the frame-level attention. Next, our new loss functions significantly help over the dual-attention and frame selection gates model by providing enhanced supervision (row 5 vs 4: p<0.0001, row 7 vs 6: p<0.005). Our RoBERTa version is also significantly better than the GloVe model (row 6 vs 4: p<0.0005, row 7 vs 5: p<0.01). Finally, employing dense captions further improves the performance via useful textual clue/keyword matching (row 8 vs 7: p<0.005).