Frustratingly Poor Performance of Reading Comprehension Models on Non-adversarial Examples
Table 7: MRR for query (MRR(Q)) and answer (MRR(A)) N-grams in Query-Aware Passage Attention layer
['Metric', 'GAR P2', 'GAR P6', 'DCN P2', 'DCN P6', 'BiDAF P2', 'BiDAF P6', 'MNR P2', 'MNR P6']
GAR, DCN, BiDAF and MNR, each compute an affinity matrix M∈Rlen(P)×len(Q), where Mij represents the similarity between the passage word at index i and query word at index j. This module aims to highlight words in the passage which are important for each query word. Again, we use the datasets P2 and P6, where the query is embedded verbatim inside the passage. We rank each N-gram of length len(Q) based on the Frobenius norm of the sub-matrix Mi:i+len(Q),: ∀i∈[0,len(P)−len(Q)] and compute the MRR of the query N-gram. For P6, the MRR is high as expected since there is no text except for the query and the answer. Interestingly, the MRR for the answer N-gram computed similarly is higher than that of the query N-gram in 3 out of 4 models on the dataset P2. This further raises the question whether the query-aware attention modules fully serve their purpose.