Classifying Constructive Commentsfootnote footnote Paper accepted for publication in First Monday. Draft version: July 24, 2020.
Table 7: Results of the length experiments. Length Corr = Length Correlation. FN Length = mean comment length of false negatives. FP Length = mean comment length of false positives.
['[BOLD] Model', '[BOLD] F1 on C3 test', '[BOLD] Length Corr.', '[BOLD] FN Length', '[BOLD] FP Length']
It can be seen that, for every model, the false positives are distributed over comments of higher length than the false negatives, indicating an over-dependence on length as a signal. This gap is smallest for CNN models, demonstrating how the the max-pooling layer in this architecture reduces its ability to overfit on comment length. The latter is measured by computing the correlation of the constructiveness score of each model’s errors with comment length. Each model shows a positive correlation, though the effect size is dramatically smaller for CNNs than the other model types due to their architectural constraints.