Frustratingly Poor Performance of Reading Comprehension Models on Non-adversarial Examples
Table 6: < UAA (Uniform Attention to Answer) and < UAQ (Uniform Attention to Query) denote the percentage of examples with less than uniform attention weight assigned to the corresponding N-gram. MRR is the Mean Reciprocal Rank of the N-gram based on the total attention weight.
['Model/Data GAR', 'Model/Data P2', '< UAA 28.64', '< UAQ 37.66', 'MRR 0.0089']
We also do another quantitative analysis wherein we consider all n-grams in the passage which have the same length as the answer. We compute a score for each n-grams as the sum of the attention weights on all the words in the n-gram. We then rank these n-grams based on this score and compute the MRR of the answer n-grams in this ranked list. The MRRs on P6 are higher due to shorter passage length.