Automatic Prediction of Discourse Connectives
Table 3: Macro-averaged F1 scores for human raters and the decomposable attention model. The three settings are: (A) All test set items; (B) Only the items for which there is a consensus among at least 2 out of 3 raters; (C) Consensus items ignoring those where either ground truth, or the rater assigned, or the model assigned label is [No connective].
['Setting', '[ITALIC] n', 'Raters (F1)', 'Model (F1)']
The macro-averaged F1 score of human raters is 23.72 which is, quite surprisingly, lower than the F1 score of the DA model, 31.80. The difference is smaller when considering majority votes on the subset of 5 714 tasks for which there is a consensus among at least 2 out of 3 raters, which results in a 30.36 F1 score for the raters. On these less ambiguous cases, the model performance also increases to 32.68. Therefore, we also evaluate the setting in which we exclude the questions for which either the ground-truth label, or the rater-assigned majority label, or the model-assigned label is [No connective].