Exploring and Distilling Cross-Modal Information for Image Captioning
Table 2: Comparisons of model complexity and speed. #Parameters are estimated. Time and Speed is measured on a single NVIDIA GeForce GTX 1080 Ti. ips stands for images per second. The symbol † denotes the result reported from original papers.
['Methods', '#Parameters', 'Train Time (h)', 'Inference Speed (ips)', 'CIDEr']
To analyze model complexity, we compare our model with (1) LSTM, which only uses a one-layer LSTM decoder [vinyals2015show] but inputs visual features at each timestep as \citeauthorlu2017knowing \shortcitelu2017knowing, (2) SoftAtt [xu2015show], which extends LSTM with visual attention, (3) Up-Down, which includes a two-layer LSTM decoder with top-down attention, (4) CT [zhu2018captioning], which is based on a transformer with six decoder blocks, and our cross-modal base model. For fair comparison, we reimplement those models using RCNN-based visual feature under the maximum log-likelihood setting. We also report obtained CIDEr scores. Our cross-modal base model is very efficient and is comparable with Up-Down in accuracy, yet 4x smaller and 2x faster. GLIED brings about a 5-point CIDEr improvement by moderate increase in parameters with even faster training and slight inference speed regression. It suggests that the cross-modal point of view and our approach benefits deep image understanding in several aspects.