RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes
Table 2: Results for simple and neural models on the test set of RecipeQA dataset.
['[EMPTY]', 'Visual Cloze', 'Textual Cloze', 'Visual Coherence', 'Visual Ordering']
For the textual cloze, the comparison between text-only and multimodal Impatient Reader models shows that the additional visual modality helps the model to understand the question better and to provide more accurate answers. While for the cloze style questions, the Impatient Reader outperforms the Hasty student, for the visual coherence and visual ordering style questions Hasty student gives way better results. This demonstrates that better neural models are needed to be able to effectively deal with this kind of questions. Some qualitative examples are provided in the supplementary material.