Spatio-Temporal Scene Graphs for Video Dialog
Table 4: Influence of Audio features on STSGR performance.
['Method', 'BLUE-1', 'BLUE-2', 'BLUE-3', 'BLUE-4', 'METEOR', 'ROUGE', 'CIDEr']
The AVSD dataset provides pre-computed audio features, encoded using the VGGish model. These are 128D features, one for every video clip, extracted from a VGG-like neural network model trained on mel-frequency spectrum of the audio associated with the video clips. (Baseline [hori2019end] MTN [zhang2018interpretable]). Both Baseline and MTN use holistic representations of the video using the I3D features. While the improvement of Baseline with the addition of VGGish features is very marginal, the transformer networks used in MTN seems to make a better use of these features. However, with STSGR, VGGish does not offer any significant advantages. This is perhaps because, our STSGR uses very fine-grained features from the frames, however the VGGish features are not directly correlated with these frame-level STSGR representations (they are computed for the video as a whole), and thus do not provide any useful details over what is already provided by STSGR. It may also suggest that perhaps we might need a better audio-visual architecture, which is outside the scope of this paper.