SentiCap: Generating Image Descriptions with Sentiments
Table 1: Summary of evaluations on captions with sentiment. Columns: sen% is the percentage of output sentences with at least one ANP; B-1 …Ciderr are automatic metrics as described in Section 5; where B-n corresponds to the BLEU-n metric measuring the co-occurrences of n-grams. Senti is the fraction of images for which at least two AMT workers agree that it is the more positive/negative sentence; Desc contains the mean and std of the 4-point descriptiveness score, larger is better. DescCmp is the percentage of times the method was judged as descriptive or more descriptive than the CNN+RNN baseline.
['Pos', 'CNN+RNN', 'sen% 1.0', 'B-1 48.7', 'B-2 28.1', 'B-3 17.0', 'B-4 10.7', 'Rouge [ITALIC] L 36.6', 'Meteor 15.3', 'Cide [ITALIC] r 55.6', 'Senti –', 'Desc 2.90±0.90', 'DescCmp –']
Results. We can see that CNN+RNN presents almost no sentiment ANPs as it is trained only on MSCOCO. SentiCap contains significantly more sentences with sentiment words than any of the three baseline methods, which is expected when the word-level regularization has taken effect. That SentiCap has more sentiment words than the two insertion baselines ANP-Replace and ANP-Scoring shows that SentiCap actively drives the flow of the sentence towards using sentimental ANPs. Sentences from SentiCap are, on average, judged by crowd sourced workers to have stronger sentiment than any of the three baselines. For positive SentiCap, 88.4% are judged to have a more positive sentiment than the CNN+RNN baseline. These gains are made with only a small reduction in the descriptiveness – yet this decrease is due to a minority of failure cases, since 84.6% of captions ranked favorably in the pair-wise descriptiveness comparison. SentiCap negative sentences are judged to have more negative sentiment 72.5% of the time. On the automatic metrics SentiCap generating negative captions outperforms all three baselines by a margin. This improvement is likely due to negative SentiCap being able to learn more reliable statistics for the new words that only appear in negative ANPs.