Fine-Grained Argument Unit Recognition and Classification
Table 1: Sentences in the selection process and final corpus size. #arg-sent: argumentative sentences. #arg-unit: argumentative units. increase in %: increase of #arg-unit compared to #arg-sent.
['#', 'topic', '#sentences', '#candidates', '#final', '#arg-sent', '#arg-unit', 'increase in %', '#non-arg']
Since we want to know, whether knowledge transfer to unknown topics is possible with few annotated topics, we limited the selection to \citeauthorStab2018b (\citeyearStab2018b) ’s eight topics (cf. This also increases comparability with related work. The topics are general enough to have good coverage in Common Crawl. They are also controversial and hence a good choice for argument mining with an expected diverse set of supporting and opposing arguments. We pre-classified the selected sentences with a sentence-level argument mining model following \citeauthorstab2018argumentext (\citeyearstab2018argumentext) and available via the ArgumenText Classify API. The API returns for each sentence (i) an argument confidence score arg_score in [0.0,1.0) (we discard sentences with arg_score<0.5), (ii) the stance on the sentence level (PRO or CON) and (iii) the stance confidence score stance_score. This information was used together with the doc_score to rank sentences for a selection in the following crowd annotation process. We convert scores for documents, arguments and stance into ranks di, ai and si. We then sum the three ranks to get an aggregated score for each sentence: aggi =di+ai+si. Sentences are divided by topic and pre-classified stance and ranked according to aggi, for each combination separately. We then go down the ranked list selecting each sentence with probability 0.5 until the target size of n=500 per stance and topic is reached; otherwise we do additional passes through the list. We adopted this probabilistic selection to ensure diversity – otherwise a long document with high relevance score at the top of the list might dominate the dataset with its sentences.