Adversarial Examples for Evaluating Reading Comprehension Systems
Table 2: Adversarial evaluation on the Match-LSTM and BiDAF systems. All four systems can be fooled by adversarial examples.
['[EMPTY]', 'Match Single', 'Match Ens.', 'BiDAF Single', 'BiDAF Ens.']
Each model incurred a significant accuracy drop under every form of adversarial evaluation. AddSent made average F1 score across the four models fall from 75.7% to 31.3%. AddAny was even more effective, making average F1 score fall to 6.7%. AddOneSent retained much of the effectiveness of AddSent, despite being model-independent. Finally, AddCommon caused average F1 score to fall to 46.1%, despite only adding common words.