Argument Component Classification for Classroom Discussions
Table 3: Results obtained with the baseline model/features and the proposed neural network models using different feature sets. Each line represents the average of a transcript-wise cross validation. Best results are in bold. ⋆, †, and ‡ indicate statistical significance at the 0.1, 0.05, and 0.01 levels respectively, compared to the model in row 3. The three right-most columns represent per-class F-score for evidence, warrants, and claims respectively.
['[BOLD] Row', '[BOLD] Models / Features', '[BOLD] Kappa', '[BOLD] Precision', '[BOLD] Recall', '[BOLD] F-score', '[ITALIC] Fe', '[ITALIC] Fw', '[ITALIC] Fc']
The statistical significance results in the table use the system in row 3 as the comparison baseline, as wLDA represents a system specifically designed for argument component classification (among other tasks). Additional statistical comparisons are provided in the text as well. This indicates that convolutional neural network models are able to extract useful features. This is not surprising, given that neural networks require a large amount of data to be trained effectively, and although random oversampling helped, we still have a limited amount of training data. Even when including additional features the two architectures show slightly different trends: CNN usually outperform LSTM, however LSTM models benefit more from the additional features. This is at least in part due to LSTMs initially having lower performance without handcrafted features. We analyzed the importance of different subsets of the online dialogue features through a feature ablation study. For CNN models, removing any subset of features resulted in a decrease in performance, except for the syntax subset in the character level CNN + wLDA + online dialogue model in both single task and multi-task settings. For LSTM models, all feature subsets contributed to increasing performance in the multi-task settings, while that was not always true for the single task models.