Whodunnit? Crime Drama as a Case for Natural Language Understanding
Table 2: Precision (pr) recall (re) and f1 for detecting the minority class (perpetrator mentioned) for humans (bottom) and various systems. We report results with cross-validation (center) and on a held-out data set (right) using the textual (T) visual (V), and auditory (A) modalities.
['[BOLD] Model', '[BOLD] Modality  [BOLD] T', '[BOLD] Modality  [BOLD] V', '[BOLD] Modality  [BOLD] A', '[BOLD] Cross-val  [BOLD] pr', '[BOLD] Cross-val  [BOLD] re', '[BOLD] Cross-val  [BOLD] f1', '[BOLD] Held-out  [BOLD] pr', '[BOLD] Held-out  [BOLD] re', '[BOLD] Held-out  [BOLD] f1']
We report precision, recall and f1 on the minority class, focusing on how accurately the models identify perpetrator mentions. The LSTM consistently outperforms the non-incremental MLP. This shows that the ability to utilize information from previous inputs is essential for this task. This is intuitively plausible; in order to identify the perpetrator, viewers must be aware of the plot’s development and make inferences while the episode evolves. The CRF is outperformed by all other systems, including rule-based PRO. In contrast to the MLP and PRO, the CRF utilizes sequential information, but cannot flexibly fuse information from different modalities or exploit non-linear mappings like neural models. We trained CRFs on audio or visual features, together with word embeddings, however these models converged to only predicting the majority class. This suggests that CRFs do not have the capacity to model complex long sequences and draw meaningful inferences based on them. PRO achieves a reasonable f1 score but does so because it achieves high recall at the expense of very low precision. The precision-recall tradeoff is much more balanced for the neural systems.