Gendered Ambiguous Pronouns Shared Task: Boosting Model Confidence by Evidence Pooling
Table 4: GREP model performance results in the Kaggle competition. Out-of-fold (OOF) error is reported on all data, i.e. gap-development, gap-validation, gap-test, and gpr-neither, as well as on gap-test explicitly for comparison against single model performance results. Since early stopping is based on OOF samples, OOF errors reported here cannot be considered as an estimate of test error. Nevertheless, stage 2 test performance benchmarks the model. †Due to a bug, the model did not fully leverage coref evidence, further gains are expected with the fixed version.
['Model', 'Dataset', 'F1 M', 'F1 F', 'F1 B', 'F1 O', 'logloss']
This performance comes very close to human performance and has almost no gender bias. As the table shows, the ensemble models achieve much larger gains in log loss as compared to F1 accuracy. This is expected since the committee of models makes more confident decisions on “easier” examples. (1) model accuracy benefits from more training data, although the gains are marginal at best (92.5 vs 93.9) given that the model was trained on approximately twice the amount of data; (2) ensembling has a similar effect as evidence pooling , i.e., models become more confident in their predictions.