Multi-modal Dense Video Captioning
Table 2: Comparison of the Feature Transformer and the Bi-directional GRU (Bi-GRU) architectures in terms of BLEU-4 (B@4), METEOR (M), and a number of model parameters. The input to all models is visual modality (I3D). The results indicate the superior performance of the Feature Transformer on all metrics. Additionally, we report the random input baseline which acts as a lower performance bound. The best results are highlighted.
['Model', 'Params. (×106)', 'Metric B@4', 'Metric M']
Firstly, we assess the selection of the model architecture. To this end, we implemented a version of our method where the transformer was replaced by Bidirectional Recurrent Neural Network with Gated Recurrent Units with attention To distil the effect of the change in architecture, the results are shown for visual-only models. Both Bi-GRU and the transformer input I3D features extracted from 64 RGB and optical flow frames (the final model inputs 24 frames). Finally, we set a lower bound for the feature performance by training a transformer model with random video features. Tab. To conclude, we observe that the feature transformer-based model is not only uses less parameters but also achieves better performance in dense video captioning task. Moreover, both method clearly surpasses the random baseline.