Integrating Stance Detection and Fact Checking in a Unified Corpus
Table 2: Performance of some stance detection models from FNC when applied to our Arabic corpus.
['[BOLD] Model', '[BOLD] document Content Used', '[BOLD] Weigh. Acc.', '[BOLD] Acc.', '[BOLD] F1 (macro)', '[BOLD] F1 ( [ITALIC] agree, disagree, discuss, unrelated [BOLD] )']
We use 5-fold cross-validation, where all claim-document pairs for the same claim are assigned to the same fold. We report accuracy, macro-average F1-score, and weighted accuracy, which is the official evaluation metric of FNC. First, we show the results when using the full document along with the claim, which is the default representation. Then, we use the best sentence from the document, i.e., the one that is most similar to the claim as measured by the cosine of their average word embeddings. This performs worse, which can be attributed to sometimes selecting the wrong sentence. Next, we experiment with using the rationale instead of the best sentence when applicable (i.e., for agree and disagree), while still using the best sentence for discuss and unrelated. This yields sizable improvements on all evaluation metrics, compared to using the best sentence (5-12 point absolute) or the full document (3-9 points absolute). We further evaluate the impact of using the rationales, when applicable, but using the full document otherwise. This setting performed best (80.2% accuracy with Athene, and 3-8 points of improvement over best+rationale), as it has access to most information: full document + rationale.