Rhetoric, Logic, and Dialectic:Advancing Theory-based Argument Quality Assessmentin Natural Language Processing
Table 10: Pearson (r) and Spearman (ρ) correlation coefficients when predicting with our BERT MTFlat trained all-domain or in-domain on the Dagstuhl data. We correlate our prediction for each dimension with its associated true label when applying mean or majority aggregation on the annotations. Scores in bold indicate best performances per domain.
['[BOLD] Domain', '[BOLD] Dimension', '[BOLD] Majority  [ITALIC] r', '[BOLD] Majority  [ITALIC] ρ', '[BOLD] Mean  [ITALIC] r', '[BOLD] Mean  [ITALIC] ρ']
For further validation, in addition to only predicting on our own corpus, AQCorpus, we, again, employ DScolor=blue!20todo: color= blue!20what is DS? I must have missed this and predict theory-based AQ scores on DS instances. In order to obtain a single gold score per annotation instance and dimension from DS, we apply two ways of aggregating the annotations provided by the authors: (1) majority vote, and (2) mean aggregation. As the BERT MTFlat modelscolor=yellow!20todo: color= yellow!20This is the problem here. have shown to perform best on many setups of the previous experiments, we reuse them – trained either on in-domain data or using data from all domains on AQCorpus – and predict AQ dimension scores on the whole DS corpus.color= yellow!20todo : color= yellow!20I need to rerun the experiments with the ST models because I employed the MT models and at this point in time, they are not discussed yet, right?color=green!20todo : color= green!20yes , I see the problem here. Generally, compared to the results on AQCorpus, the performance drops. This can be attributed to the domain-transfer. While DS also consist of web debate arguments, they consist of individual isolated arguments and not – as our corpus – of more complex texts. Training on our joint training set across all domains yields the highest correlation scores and generally, the annotations related to overall argument quality and reasonableness correlate best with our scores. To the best of our knowledge, we are the first to have a trained model in-place which is able to predict scores for fine-grained argument quality dimensions.