Consistent Automated Claim Detection
Table 3: From 7 categories to binary claim vs “not a claim” classification. N = number of sentences annotated by majority. (1) Personal experience, (2) Quantity in the past or present, (3) Correlation or causation, (4) Current laws or rules of operation, (5) Prediction, (6) Other type of claim, (7) Not a claim.
['[BOLD] Claim', '[BOLD] Non-claim', '[BOLD] Omitted', '[ITALIC] α', 'N']
To solve the disagreements we reframe the task as binary classification by grouping some categories together. Another reason for the binary instead of multi-class classification is that the end user application is binary, i.e. a sentence either makes it to the excerpted list of claims or not. The binary classification system still captures the expert factcheckers input by using the granular positive claim categories during annotation. Most importantly it achieves dataset robustness by reducing the annotator disagreement. The are no negative sides to this binary re-formulation for this use case. The claims are routed to the relevant factchecking team or a dataset based on their topic (e.g. healthcare or immigration) and not claim type. ‘Quantity’ was selected as the positive class because it is the majority category of claims (received most annotations). It’s also the category that Full Fact most frequently writes about. We established this after annotating 800 of their claims. The categories of ‘Personal experience’ and ‘Prediction’ were excluded for ease of modelling because they might contain quantities outside of the ‘Quantity’ claim class. Here ‘Other type of claim’ is not in the positive class for two reasons. First of all there is a lot of disagreement between it and the ‘Not a claim’ class. Secondly, the kinds of claim in the ’Other’ section - voting records, quotes, statements about public opinion polls, are less frequently written about by Full Fact. We are aware that this choice encodes the peculiarity of one organisation at one point in time into our model and needs to be re-considered if, say, an organisation wishing to use the model specialises in voting records.