Robust Machine Comprehension Models via Adversarial Training
Table 2: F1 performance of the BSAE model trained on datasets with different distractor placement strategies.
['Training', 'AddSent', 'AddSentPrepend', 'Average']
We also conducted experiments studying the effect of different distractor placement strategies on the trained models’ robustness. The BSAE model was trained on 4 variations of AddSentDiverse-augmented training set, with the only difference between them being the location of the distractor within the context: InsFirst, where the distractor is prepended, InsLast, where the distractor is appended, InsMid, where the distractor is inserted in the middle and InsRandom, where the distractor is randomly placed. The retrained models are tested on AddSent and AddSentPrepend, whose only difference is where the distractor is located. It is clear that when trained under InsFirst and InsLast, the model only performs well on test sets created by a similar distractor placement strategy, indicating that they are exploiting superficial trends instead of learning to process the semantics. It is also shown that InsRandom gives optimal performance on both evaluation datasets. Further investigations regarding distractor placement can be found in the appendix. A bidirectional deep learning model, trained in a supervised setting, should be able to jointly learn X and Y. Thus, at test time, when given a paragraph of n sentences, the model can obtain the probability that the sentence sa is the distractor, Psa, by computing P(X= a)+P(Y=n−a). Ideally, we want the distribution of Psa to be uniform, as that means the model is not biased towards discarding any sentence as the distractor based on location. The actual distributions of Psa under different distractor-insertion strategies are displayed in Fig. We pick these n as they are typical lengths of contexts within the SQuAD dataset (the complete distribution of paragraph lengths in the SQuAD training set is shown in Fig. We see that under random insertion, the distribution is very close to uniform. Note that if we were to aggregate n and plot Psa for n≤3, 5 and 7, as shown in Fig.