Why Comparing Single Performance Scores Does Not Allow to Draw Conclusions About Machine Learning Approaches
Table 2: The same BiLSTM-CRF approach was evaluated twice under Evaluation 2. The threshold column depicts the average difference in percentage points F1-score for statistical significance with 0.04
['[BOLD] Task', '[BOLD] Spearman  [ITALIC] ρ', '[BOLD] Threshold  [ITALIC] τ', '[BOLD] % significant', 'Δ( [ITALIC] dev)95', 'Δ( [ITALIC] test)95', 'Δ( [ITALIC] test) [ITALIC] Max']
For all tasks, we observe small Spearman’s rank correlation ρ between the development and the test score. The low correlation indicates that a run with high development score doesn’t have to yield a high test score. The value 3.68 for the ACE 2005 - Events tasks indicates that, given two models with the same performance on the development set, the test performance can vary up to 3.68 percentage points F1-score (95% interval).