Trouble on the Horizon:Forecasting the Derailment of Online Conversations as they Develop
Table 1: Comparison of the capabilities of each baseline and our CRAFT models (full and without the Context Encoder) with regards to capturing inter-comment (D)ynamics, processing conversations in an (O)nline fashion, and automatically (L)earning feature representations, as well as their performance in terms of (A)ccuracy, (P)recision, (R)ecall, False Positive Rate (FPR), and F1 score. Awry is the model previously proposed by Zhang et al. (2018) for this task.
['[BOLD] Model', 'Capabilities  [BOLD] D', 'Capabilities  [BOLD] O', 'Capabilities  [BOLD] L', 'Wikipedia Talk Pages  [BOLD] A', 'Wikipedia Talk Pages  [BOLD] P', 'Wikipedia Talk Pages  [BOLD] R', 'Wikipedia Talk Pages  [BOLD] FPR', 'Wikipedia Talk Pages  [BOLD] F1', 'Reddit CMV  [BOLD] A', 'Reddit CMV  [BOLD] P', 'Reddit CMV  [BOLD] R', 'Reddit CMV  [BOLD] FPR', 'Reddit CMV  [BOLD] F1']
Results. First, we find that unsurprisingly, accounting for full conversational context is indeed helpful, with even the simple online baselines outperforming the fixed-window baselines. On both datasets, CRAFT outperforms all baselines (including the other online models) in terms of accuracy and F1. Furthermore, although it loses on precision (to CRAFT − CE) and recall (to Cumulative BoW) individually on the Wikipedia data, CRAFT has the superior balance between the two, having both a visibly higher precision-recall curve and larger area under the curve (AUPR) than the baselines This latter property is particularly useful in a practical setting, as it allows moderators to tune model performance to some desired precision without having to sacrifice as much in the way of recall (or vice versa) compared to the baselines and pre-existing solutions.