Spatio-Temporal Scene Graphs for Video Dialog
Table 1: Ablation study on STSGR using AVSD@DSTC7 dataset.
['Method', 'BLUE-1', 'BLUE-2', 'BLUE-3', 'BLUE-4', 'METEOR', 'ROUGE', 'CIDEr']
Ablation Study: We analyze several key components: (i) shuffling in the Transformer structure, (ii) visual and semantic graph, (iii) ROI Recrop on the union bounding boxes, and (iv) temporal aggregation. From the table, we see that Graph Attention Network (GAT), which is used to produce the visual scene graph, is important to aggregate information from neighboring nodes (e.g., improving CIDEr from 1.125 to 1.265), while EdgeConv, used in the semantic graph, offers some improvement (e.g., CIDEr from 1.244 to 1.265). Moreover, the use of shuffling in the multi-head Transformer structure boosts the performance significantly (from 1.208 to 1.265 for CIDEr). We can also conclude that union bounding boxes, semantic labels, and inter-graph reasoning contribute to stabilize the generation performance. Overall, by adopting all these key components, the full model outperforms all the ablations.