Distinctive-attribute Extraction for Image Captioning
Table 4: Results of experiments with differently constructed vocabularies with stemming. (a) and (b) represent results with stemming and without stemming, respectively. The prediction performance in F1 decreases from Vocab5 to Vocab11, and the best captioning performance in CIDEr is obtained at Vocab7 in both (a) and (b)
['[EMPTY]', 'Vocab5', 'Vocab6', 'Vocab7', 'Vocab8', 'Vocab9', 'Vocab10', 'Vocab11']
We set seven different IDF threshold values, thIDF, from 5 to 11. Vocabi(i∈{5,6,…,11}), (16) Vocabi={w|IDF(w)>i, i=thIDF}. (17) The vocabulary contains only the words whose IDF is bigger than thIDF. For example, the number of words in Vocab5 is 276 out of total 5,663 words after stemming in reference captions. Semantic information of the images are extracted corresponding to this vocabulary, and we use them to learn the proposed prediction model. The performance, macro-averaged F1, of the prediction model evaluated by test splits is shown in the third row. The lower the thIDF, that is, the vocabulary is composed of the more frequent words, provides the better prediction performance. Each extracted distinctive-attribute is fed into SCN-LSTM to generate a caption, and the evaluation result, CIDEr, is shown in the fourth row. The CIDErs increase from Vocab5 to Vocab7, and then monotonically decrease in the rest. In other words, the maximum performance is derived from Vocab7 to 0.996. The vocabulary size and the prediction performance are in a trade-off in this experiment. With the high thIDF value, captions can be generated with various vocabularies, but the captioning performance is not maximized because the performance of distinctive-attribute prediction is relatively low. The captioning performance is highest at Vocab7. The value was 0.911, which is lower than the maximum value of the experiments with stemming. When stemming is applied, the distinctiveness and significance of a word can be better expressed because it is mapped to the same word even if the tense and form are different. The size of vocabulary required to achieve the same performance is less when stemming is applied. It means that the number of parameters needed for the captioning model is small and the computational complexity is low.