DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Table 4: Performance of the different models on our development and test set, in terms of Exact Match (EM), and numerically-focused F1 (§5). Both metrics are calculated as the maximum against a set of gold answers.
['[BOLD] Method', '[BOLD] Dev EM', '[BOLD] Dev F1', '[BOLD] Test EM', '[BOLD] Test F1']
Most notably, all models perform significantly worse than on other prominent reading comprehension datasets, while human performance remains at similar high levels. For example, BERT, the current state-of-the-art on SQuAD, drops by more than 50 absolute F1 points. This is a positive indication that DROP is indeed a challenging reading comprehension dataset, which opens the door for tackling new and complex reasoning problems on a large scale.