Adversarial attacks againstFact Extraction and VERification
Table 4: Breakdown of the potency of the three classes of transformation for each model. For the models with oracle information retrieval, the FEVER Score is equal to the label accuracy.
['[BOLD] Model', '[BOLD] Transformation', '[BOLD] Accuracy  [BOLD] Before', '[BOLD] Accuracy  [BOLD] After', '[BOLD] Accuracy  [BOLD] Delta', '[BOLD] FEVER Score  [BOLD] Before', '[BOLD] FEVER Score  [BOLD] After', '[BOLD] FEVER Score  [BOLD] Delta']
Considering the performance of the systems we test, we also present a leader-board ranked by the resilience (average FEVER score for all breaker methods). While the Papelo system was ranked 4th on the FEVER shared task, this model performs the best in this evaluation. This may arise from the fact that both teams make use of the development data that we used to generate our claims differently (i.e. the pretrained UCL model we used incorporated development data to increase the size of the training set whereas other models did not). As it is not possible to control how this data is used in the FEVER2.0 shared task, our method for generating claims from transforming existing instances may perform better (i.e. have more potency) when applied to instances that have not previously been released to the public.