ExCL: Extractive Clip Localization Using Natural Language Descriptions
Table 1: Clip Localization Accuracy at IoU = {0.3, 0.5, 0.7} for TACoS, Charades-STA and ActivityNet. Here ExCL-clf represents the classification loss model and ExCL-reg represents the regression loss model. ExCL-{clf/reg} 1-m refer to models run without a video LSTM encoder, while ExCL-{clf/reg} 2-m include the video LSTM. m=a,b,c refer to MLP, tied LSTM and conditioned LSTM span predictor networks respectively.
['IoU', 'TACoS 0.3', 'TACoS 0.5', 'TACoS 0.7', 'Charades-STA 0.3', 'Charades-STA 0.5', 'Charades-STA 0.7', 'ActivityNet 0.3', 'ActivityNet 0.5', 'ActivityNet 0.7']
We find that across all datasets, models without recurrent architectures (ExCL 1-a) perform significantly worse, demonstrating the importance of temporal context provided by LSTMs in either video encoder (ExCL 2-a) or span predictors (ExCL 1-b). With ExCL 2-a we see that if the visual context is captured well using recurrent architectures in the video LSTM, then even with a MLP span predictor we get a significant improvement in performance. Furthermore, if we add LSTM span predictors along with the video LSTM (ExCL 2-{b,c}) we obtain an additional boost in performance. However, without a recurrent visual encoder, a recurrent span predictor is essential to capture both uni-modal and cross-modal interactions (ExCL 1-{b, c}). When comparing the regression learning objective with classification, we do not notice a substantial gain in performance thereby indicating that not much information is lost if the continuous nature of the labels is not considered. In terms of span predictors, tied LSTM generally performs well across all datasets, and any difference with conditioned LSTM is negligible. While Xu et al. We also find TACoS to be a significantly more challenging benchmark, similar to Ge et al.