Canonical Correlation Inference for Mapping Abstract Scenes to Text
Table 2: Scene description evaluation results on the test set, comparing the systems from Ortiz et al. to our CCA inference algorithm (the first six results are reported from the Ortiz et al. paper). The CCA result uses m=120 and η=0.05, tuned on the development set. See text for details about each of the first six baselines.
['[EMPTY]', 'system', 'BLEU', 'METEOR']
\citeauthor ortiz2015learning \shortciteortiz2015learning partially measure the success of their system by comparing BLEU and METEOR scores of their different systems while using the descriptions given in the dataset as a reference set. MLBL: mutlimodal log-bilinear model, implementation of \citeauthorkiros2014multimodal \shortcitekiros2014multimodal. Image: a system that for every new image, queries the set of training images for the most similar one, and returns a random description of that training example. Keyword: system that annotates every image with keywords that most probably describe it and then do a search query against all training data descriptions, returning the description that is closest (in terms of TF-IDF similarity) to the keywords. Template: system that uses templates inferred from dependency parses of the training data descriptions. A set of templates is discovered and a classifier that associates images with templates is trained. SMT: Ortiz et al. system first selects pairs of clipart objects that are important enough to be described by solving an integer linear programming problem, creates a “visual encoding” using visual dependency grammar and finally uses a phrase-based SMT engine to translate the latter to proper sentences.