Mean Box Pooling: A Rich Image Representation and Output Embedding for the Visual Madlibs Task
Table 5: Comparison between our Embedded CNN+LSTM approach that computes the similarity between input and candidate answers in the embedding space, and the plain CNN+LSTM original approach from [Yu et al.(2015)Yu, Park, Berg, and Berg]. Since the accuracies of CNN+LSTM [Yu et al.(2015)Yu, Park, Berg, and Berg] are unavailable for two categories, we report average over 10 categories in this case. Results in %.
['[BOLD] Easy Task', '[BOLD] Easy Task Embedded', '[BOLD] Easy Task Ask Your', '[BOLD] Easy Task CNN+LSTM', '[BOLD] Hard Task Embedded', '[BOLD] Hard Task CNN+LSTM']
We hypothesize that, likewise to nCCA, in order to choose a completion of the prompt sentence out of four candidates, the comparison between the candidate completions should be directly done in the output embedding space. Note that, here we feed the sentence prompt to LSTM even though it is fixed per category. Our “Embedded CNN+LSTM” outperforms other methods on both tasks confirming our hypothesis. (on the 10 categories that the results for CNN+LSTM are available it achieves 49.8% accuracy on the easy task, which is 2.1 percentage points higher than CNN+LSTM).