Adversarial Filters of Dataset Biases
Table 7: AFLite hyperparameters used for running for in-distribution benchmark estimation on different datasets. m denotes the size of the support of the expectation in Eq. (4), t is the training set size for the linear classifiers, k is the size of each slice, and τ is an early-stopping filtering threshold. For ImageNet, we set n=640K and hence do not need to control for τ. In every other setting, we set τ as above, and hence do not need to control for n.
['[EMPTY]', 'Synthetic', 'SNLI', 'MultiNLI', 'QNLI', 'ImageNet']
At each filtering phase, we train models (linear classifiers) on m different random partitions of the data, and collect their predictions on their corresponding test set. For each instance i, we compute its predictability score as the ratio of the number of times its label yi is predicted correctly, over the total number of predictions for it. We rank the instances according to their predictability score and use the greedy slicing strategy of removing the top-k instances whose score is not less than the early-stopping threshold τ. We repeat this process until fewer than k instances pass the τ threshold in a filtering phase or fewer than n instances remain. Target dataset size, n and the early stop filtering threshold τ are interdependent, as the predictability score threshold determines what examples to keep, which in turn influences the desired size of the dataset, n. For ImageNet, we set n=640K and do not control for τ. We use much larger values for t and k for ImageNet than in all NLP experiments, where the use of powerful language representations such as RoBERTa allows us to get reasonable performance even with smaller training sets; ImageNet does not offer any such benefits due to pretrained representations.