Why Comparing Single Performance Scores Does Not Allow to Draw Conclusions About Machine Learning Approaches
Table 5: 95% percentile of Δ(test) after averaging.
['[BOLD] Task', 'Δ( [ITALIC] test)95 [BOLD]  for  [ITALIC] n scores  [BOLD] 1', 'Δ( [ITALIC] test)95 [BOLD]  for  [ITALIC] n scores  [BOLD] 3', 'Δ( [ITALIC] test)95 [BOLD]  for  [ITALIC] n scores  [BOLD] 5', 'Δ( [ITALIC] test)95 [BOLD]  for  [ITALIC] n scores  [BOLD] 10', 'Δ( [ITALIC] test)95 [BOLD]  for  [ITALIC] n scores  [BOLD] 20']
For increasing n the value Δ(test)95 decreases, i.e. the mean score becomes more stable. However, for the CoNLL 2003 NER- En task we still observe a difference of 0.26 percentage points F1-score between the mean scores for n=10. For the ACE 2005 Events dataset, the value is even at 1.39 percentage points F1-score.