Empirical Autopsy of Deep Video Captioning Frameworks
Table 6: Percentage improvement achieved in BLEU (B-4), METEOR (M), CIDEr (C), and ROUGEL (R) metrics when mean pooled visual features are replaced with temporally encoded features of corresponding networks.
['[EMPTY]', 'B-4 (%)', 'M (%)', 'C (%)', 'R (%)']
As can be observed, there is significant improvement in the model performance across all metrics and all networks with the temporal encoding. The largest performance gain results in the case of C3D. We conjecture that a major reason behind this phenomenon is that there are always less number of clips as compared to the number of frames in videos. C3D exploits clips which reduces the number of unit data samples containing distinct pieces of information for the task at hands. The temporal encoding strategy is able make up for this discrepancy. Moreover, spatial feature capturing with a 2D-CNN followed by temporal encoding result in more discriminative video-level features as compared to the spatio-temporal features of 3D-CNNs.