Argument Identification in Public Comments from eRulemaking
Table 7. Claim classification of argument types.
['[EMPTY]', 'LR+w2v claim-neutral', 'LR+w2v claim-neutral', 'LR+w2v claim-neutral', 'LR+w2v supp v. opp', 'LR+w2v supp v. opp', 'LR+w2v supp v. opp', 'LR+w2v claim+neutral', 'LR+w2v claim+neutral', 'LR+w2v claim+neutral', 'LR+w2v claim+ensemble', 'LR+w2v claim+ensemble', 'LR+w2v claim+ensemble', 'fastText claim-neutral', 'fastText claim-neutral', 'fastText claim-neutral', 'fastText supp v. opp', 'fastText supp v. opp', 'fastText supp v. opp', 'fastText claim+neutral', 'fastText claim+neutral', 'fastText claim+neutral']
The overall performance is 0.60-0.64 F1, with likely opposition, legal challenge, and seeks exclusion performing the best, while lacks flexibility, too narrow, and too broad are worst. Somewhat surprisingly we see stronger overall performance on support than oppose, with a macro-F1 of 0.85 for support and 0.65-0.70 for oppose. On an argument level, likely support and likely opposition both do well, while too broad, too narrow, and lacks flexibility perform the worst. In all of the above except claim identification, we assume we know which sentences are argumentative, or what their stance is. However, in a real-world setting we do not. As expected, we achieve a significantly lower macro-F1 as the arguments’ classification performance drops, while the neutral class remains in the 0.90s. Least affected are conflicting interests, burdensome, and requests clarification. Finally, taking advantage of the proposed hierarchical structure of the argument taxonomy, we train the claim identification, stance detection, and separate stance-based claim classification models as before, where we have knowledge of the true argumentative sentences and stance, and augment the claim+neutral data with each of their output probabilities. We use this augmented data to build an ensemble claim classification model that in addition to the original feature space has the predicted probabilities of each of the above models as input features. This model achieves a macro-F1 improvement of 4% over claim+neutral to 0.5.