Classifying Constructive Commentsfootnote footnote Paper accepted for publication in First Monday. Draft version: July 24, 2020.
Table 6: Feature and domain adaptation results. Each cell shows the F1 score with the given model and train/test setting.
['[BOLD] Model', '[BOLD] TRAIN  [BOLD] TEST', '[BOLD] C3 train  [BOLD] C3 test', '[BOLD] C3 train  [BOLD] NYT +YNACC*', '[BOLD] NYT +YNACC*  [BOLD] C3 test']
The first results column shows F1 scores for different models when we trained on C3 train and tested on C3 test. Among the feature-based models, our four length features are the best predictors of constructiveness. This is not a surprise given the skewed distribution of length in constructive and non-constructive comments (see The text-quality and all features also demonstrate comparable performance. Perspective’s The second set of experiments was conducted to examine how transferable the learned models are on different datasets and whether the models are capturing general characteristics of constructiveness or topic-specific characteristics. C3 contains comments posted on articles from a Canadian national newspaper, which are likely to discuss Canadian issues and politics, whereas NYT and YNACC * contain comments posted on The New York Times and Yahoo News articles, respectively, which are more likely to address American issues and politics. We examined how features learned on one dataset perform on the other. The second results column (C3 train and NYT + YNACC* test) and the third results column (NYT + YNACC While length is the most important feature in the single domain context, when we move to a new context where the training and test sets differ, its relative importance decreases. In the domain transfer context, text quality and lexical features play an important role. In case of the deep models, the performance drops markedly when we move to a new context, but they are still the best performing models when trained on C3 train and tested on NYT + YNACC * When trained on NYT + YNACC * and tested on C3 test, CNNs and BERT perform poorly. This is perhaps because the training data itself is noisy. Recall that NYT contributed the positive cases (constructive) and YNACC* the negative instances. Drawing positive and negative samples from different sources is never ideal, and likely explains the modeling mismatch. For instance, the readability score from the text quality features is a function of length of the comment, calculated using the SMOG index formula (mc_laughlin_smog_1969), as shown in Equation (1). 1.043×√words with polysyllables×30% \ It is also vulnerable to adversaries who could attempt to write long low-quality comments in order to fool the models into classifying their comments as constructive.