Adversarial attacks againstFact Extraction and VERification
Table 2: Summary of adversarial attack potency on three models using the FEVER development data split. For the Before and After columns, the figures are reported on the subset of the data used for the rule based transformation. We also report accuracy on the entire development split reported in Thorne et al. [28] (9999 instances in total, Dev). For the case of the oracle evidence retrieval component, FEVER Score is equal to accuracy.
['[BOLD] Model', '[BOLD] Accuracy  [BOLD] Dev', '[BOLD] Accuracy  [BOLD] Before', '[BOLD] Accuracy  [BOLD] After', '[BOLD] Accuracy  [BOLD] Delta', '[BOLD] FEVER Score  [BOLD] Dev', '[BOLD] FEVER Score  [BOLD] Before', '[BOLD] FEVER Score  [BOLD] After', '[BOLD] FEVER Score  [BOLD] Delta']
65 rule-based transformations were executed to the data from the development set. There were 2708 instances that matched the regular expression patterns defined in these rules, which were used to generate 29495 novel adversarial instances. In the new set of claims that were generated, only 15% were labeled as NotEnoughInfo (as the negating rules were only applied to claims which were Supported or Refuted) which would preclude comparison against existing results as the FEVER evaluation which assumes balanced class distribution. We balance our data by discarding instances at random from the majority classes.