Modeling Subjective Assessments of Guilt in Newspaper Crime Narratives
Table 2: MSE for predicting guilt ratings for the Reader perception and Author belief questions. The models themselves are defined in Section 4. We report the mean and standard derivation values from 20 different runs. Bold denotes the best performance.
['[EMPTY]', '[ITALIC] Reader perception', '[ITALIC] Author belief']
Below are some additional details on these comparisons (‘AB’ = Author belief; ‘RP’ = Reader perception. Our statistical test here is the Wilcoxon signed-rank test.) BERT with the CLS token does not improve performance compared to a simple mean baseline (p=0.440 for RP and p=0.996 for AB), while BERT with mean-pooling achieves better performance compared to the mean baseline (p<0.001 for RP and p=0.004 for AB). The differences between using mean pooling and the CLS token are significant (p=0.004 for RP and p<0.001 for AB). When using both the genre pretraining and the token supervision, mean pooling is significantly better than using the CLS token (p=0.001 for RP and p<0.022 for AB). Overall, a mean pooling model that makes use of genre pretraining as well as span-level supervision achieves the best performance, significantly outperforming other models (p<0.001 for RP and p=0.026 for AB when comparing with the mean baseline; p=0.001 for RP and p=0.124 for AB with genre pretraining; and p<0.001 for RP and p=0.022 for AB with joint supervision). Neither genre pretraining (p=0.645 for RP and p=0.455 for AB) nor span-level supervision (p=0.002 for RP and p=0.206 for AB) alone can improve performance a lot in comparison to the mean baseline (only joint supervision for RP is significant).