Multi-modal Dense Video Captioning
Table 3: The performance of the proposed MDVC framework with different input modalities (V-visual, A-audio, S-speech) and feature fusion approaches: probability averaging and concatenation of two fully-connected layers (Concat. + 2 FC). Also, we report the comparison between audio-visual MDVC with visual-only MDVC with similar model capacities (2 FC).
['Modality V', 'Modality A', 'Modality S', 'Fusion', 'Params. (×106)', 'Metric B@4', 'Metric M']
Secondly, we evaluate the contribution of different modalities in our framework. Tab. Specifically, averaging of the output probabilities and concatenation of the outputs of all modalities and applying two fully connected (FC) layers on top. We observe that audio-only model has the worst performance, followed by the visual only model, and the combination of these two. Moreover, the concatenation and FC layers result in better performance than averaging. To further assess if the performance gain is due to the additional modalities or to the extra capacity in the FC layers, we trained a visual-only model with two additional FC layers. The results indicate that such configuration performs worse than any bi-modal setup. Overall, we conclude that the final model with all three modalities performs best among all tested set-ups, which highlights the importance of multi-modal setting in dense video captioning task.