Riposte! A Large Corpus of Counter-Arguments
Table 3: The average Jaccard’s similarity scores between CAs for a single argument for each fallacy type.
['[BOLD] Criteria Score', '0.61', '0.17', '0.35', '0.36', '[BOLD] Total 0.24']
How similar are the CAs across annotators? One design decision when building Riposte! was that with more annotators , we could collect a wide variety of diverse CAs for a single-argument regardless of the fallacy type. We first calculate the similarity of the CAs across annotators for a single argument. We then calculate the average Jaccard similarity score for all combinations of CAs per unique argument and average over all arguments.