Selection Bias Explorations and Debias Methods forNatural Language Sentence Matching Datasets
Table 3: Evaluation Results with the testing set of QuoraQP. We report the accuracy scores and “%” is omitted.
['[BOLD] Method', '[BOLD] Biased Eva', '[BOLD] Debiased Eva']
From the accuracy of the method Leakage, we can see that although the influence isn’t completely eliminated, the evaluation result of Debiased Eva is less impacted by the bias pattern in the original distribution. This makes the results more reliable for evaluations. The reason why in the Leakage method the bias could not be completely eliminated is that we cannot estimate P(S=y|l) perfectly. A minor error of P(S=y|l) may result in a significant difference in the weight especially when the probability is close to zero, since the multiplicative inverse is used.