The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
Table 2: Combining image and text in neural architectures improves their ability to predict the next image or dialogue in comics narratives. The contextual information present in preceding panels is useful for all tasks: the model that only looks at a single panel (NC-image-text) always underperforms its context-aware counterpart. However, even the best performing models lag well behind humans.
['[BOLD] Model', '[BOLD] Text Cloze  [ITALIC] easy', '[BOLD] Text Cloze  [ITALIC] hard', '[BOLD] Visual Cloze  [ITALIC] easy', '[BOLD] Visual Cloze  [ITALIC] hard', '[BOLD] Char. Coheren.']
On text cloze, the image-text model dominates those trained on a single modality. However, text is much less helpful for visual cloze than it is for text cloze, suggesting that visual similarity dominates the former task. Having the context of the preceding panels helps across the board, although the improvements are lower in the hard setting. There is more variation across the models in the easy setting; we hypothesize that the hard case requires moving away from pretrained image features, and transfer learning methods may prove effective here. Differences between models on character coherence are minor; we suspect that more complicated attentional architectures that leverage the bounding box locations bix are necessary to “follow” speech bubble tails to the characters who speak them.