Adversarial attacks againstFact Extraction and VERification
Table 3: Effect of adversarial attacks on the evidence retrieval component of the pipelines considering sentence-level accuracy of the evidence.
['[BOLD] Model', '[BOLD] Precision  [BOLD] Before', '[BOLD] Precision  [BOLD] After', '[BOLD] Precision  [BOLD] Delta', '[BOLD] Recall  [BOLD] Before', '[BOLD] Recall  [BOLD] After', '[BOLD] Recall  [BOLD] Delta', '[BOLD] F1  [BOLD] Before', '[BOLD] F1  [BOLD] After', '[BOLD] F1  [BOLD] Delta']
We observe that in the oracle environment (using manually labeled evidence), both the ESIM and Decomposable Attention models for natural language inference suffer a stark decrease in accuracy when making predictions on the adversarial examples. The decrease is less pronounced when considering the full pipeline (i.e. with a non-perfect evidence retrieval component), which reduces the upper bound for the FEVER Score and accuracy.