From Captions to Visual Concepts and Back
Table 3: Average precision (AP) and Precision at Human Recall (PHR) [capeval2015] for words with different parts of speech (NN: Nouns, VB: Verbs, JJ: Adjectives, DT: Determiners, PRP: Pronouns, IN: Prepositions). Results are shown using a chance classifier, full image classification, and Noisy OR multiple instance learning with AlexNet [krizhevskyNIPS12] and VGG [simonyan14very] CNNs.
['[EMPTY]', 'Average Precision NN', 'Average Precision VB', 'Average Precision JJ', 'Average Precision DT', 'Average Precision PRP', 'Average Precision IN', 'Average Precision Others', 'Average Precision All', 'Precision at Human Recall NN', 'Precision at Human Recall VB', 'Precision at Human Recall JJ', 'Precision at Human Recall DT', 'Precision at Human Recall PRP', 'Precision at Human Recall IN', 'Precision at Human Recall Others', 'Precision at Human Recall All']
Average Precision (AP) and Precision at Human Recall (PHR) We report two baselines. The first (Chance) is the result of randomly classifying each word. The second (Classification) is the result of a whole image classifier which uses features from AlexNet or VGG CNN [krizhevskyNIPS12, simonyan14very]. These features were fine-tuned for this word classification task using a logistic regression loss. In fact, we observe the largest improvement for nouns and adjectives, which often correspond to concrete objects in an image sub-region. Results for both classification and MIL NOR are lower for parts of speech that may be less visually informative and difficult to detect, such as adjectives (e.g., few, which has an AP of 2.5), pronouns (e.g., himself, with an AP of 5.7), and prepositions (e.g., before, with an AP of 1.0). In comparison words with high AP scores are typically either visually informative (red: AP 66.4, her: AP 45.6) or associated with specific objects (polar: AP 94.6, stuffed: AP 74.2).