The NarrativeQA Reading Comprehension Challenge
Table 5: Experiments on summaries. Higher is better for all metrics. Sections 4.1 and 4.2 explain the IR and neural models, respectively.
['[BOLD] Model', '[BOLD] Model', '[BOLD] Validation / Test  [BOLD] Bleu-1', '[BOLD] Validation / Test  [BOLD] Bleu-4', '[BOLD] Validation / Test  [BOLD] Meteor', '[BOLD] Validation / Test  [BOLD] Rouge-L', '[BOLD] Validation / Test  [BOLD] MRR']
Each chunk contains 200 tokens. Higher is better for all metrics. Reading comprehension of summaries is similar to a number of previous reading comprehension tasks where questions were constructed based on the context document. However, plot summaries tend to contain more intricate event time lines and a larger number of characters, and in this sense, are more complex to follow than news articles or paragraphs from Wikipedia. Given the retrieved chunks, we experimented with several neural models using them as context. While is does slightly better on the other metrics, it clearly fails to make use of the retrieved context to gain a distinctive margin over the no-context Seq2Seq model. Increasing the number of retrieved chunks, and thereby recall of possibly relevant parts of the story, had only a minor positive effect. The span prediction model—which here also uses selected chunks for context—does especially poorly in this setup. While this model provided the best neural results on the summaries task, we suspect that its performance was particularly badly hurt by the fact that there is so little lexical and grammatical overlap between the source of the questions (summaries) and the context provided (stories). As with the AS Reader, we observed no significant differences for varying number of chunks.