The NarrativeQA Reading Comprehension Challenge
Table 6: Experiments on full stories. Each chunk contains 200 tokens. Higher is better for all metrics. Sections 4.1 and 4.2 explain the IR and neural models, respectively. Note that the human scores are based on answering questions given summaries, same as in Table 5.
['[BOLD] Model', '[BOLD] Model', '[BOLD] Validation / Test  [BOLD] Bleu-1', '[BOLD] Validation / Test  [BOLD] Bleu-4', '[BOLD] Validation / Test  [BOLD] Meteor', '[BOLD] Validation / Test  [BOLD] Rouge-L', '[BOLD] Validation / Test  [BOLD] MRR']
As expected (and desired), we observe a decline in performance of the span-selection oracle IR model, compared with the results on summaries. This is unsurprising as the questions were constructed on summaries and confirms the initial motivation for designing this task. As previously, we considered all spans of a given length across the entire story for this model. For short answers of one or two words—typically main characters in a story—the candidate, i.e. the closest span to the reference answer, is easily found due to being mentioned throughout the text. For longer answers it becomes much less likely, compared to the summaries, that a high-scoring span can be found in the story. Note that this distinguishes NarrativeQA from many of the reviewed datasets.