Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA
Table 2: EM scores on 4 adversarial evaluation settings after training on Add4Docs-Rand. ‘-R’ and ‘-P’ represent random insertion and prepending. A4D and A8D stands for Add4Docs and Add8Docs adv-dev sets.
['[EMPTY]', 'A4D-R', 'A4D-P', 'A8D-R', 'A8D-P']
In order to test the robustness of the adversarially-trained models against new adversaries, we additionally evaluate them on dev sets with varying numbers of adversarial documents and a different adversary placement strategy elaborated in Sec. When the number of adversarial documents per supporting document with answer is increased to eight, all four models’ performance drops by more than 1 points, but again the 2-hop model, with or without supporting-fact supervision, continues to outperform its single-hop counterpart.