FAKTA: An Automatic End-to-End Fact Checking System
Table 2: FAKTA full pipeline Results on FEVER.
['[BOLD] Model', '[BOLD] Settings', '[BOLD] F1( [ITALIC] SUP/ [ITALIC] REF/ [ITALIC] NEI)', '[BOLD] F1( [ITALIC] Macro)', '[BOLD] Acc.']
Lines 1–3 show the results for all three SUP, REF, and NEI labels (3lbl) and Randomly Sampled (RS) documents from Wikipedia for the NEI label. We label claims as NEI if the most relevant document retrieved has a retrieval score less than a threshold, which was determined by tuning on development data. Line 1 is the multi-layer perceptron (MLP) model presented in Riedel et al. Lines 2–3 are the results for our system when using Lucene (L) and Google API (G) for document retrieval. The results show that our system achieves the highest performance on both F1(Macro) and accuracy (Acc) using Google as retrieval engine. We repeat our experiments when considering only SUP and REF labels (2lbl) and the results are significantly higher than the results with 3lbl (Lines 4–5).