Selection Bias Explorations and Debias Methods forNatural Language Sentence Matching Datasets
Table 1: The accuracy scores of predicting the label with unlexicalized features, leakage features, and advanced graph-based features and the relative improvements. Result with ∗ is from Bowman et al. (2015). Results with † are from Williams et al. (2018). Result with ‡ is from Wang et al. (2017). Result with ⋄ is from Shen et al. (2018). Result with ⊤ is from Baudiš et al. (2016). Other results are based on our implementations. “%” is omitted.
['[BOLD] Method', '[BOLD] SNLI', '[BOLD] MultiNLI  [BOLD] Matched', '[BOLD] MultiNLI  [BOLD] Mismatched', '[BOLD] QuoraQP', '[BOLD] MSRP', '[BOLD] SICK  [BOLD] NLI', '[BOLD] SICK  [BOLD] STS', '[BOLD] ByteDance']
Based on the new problem definition, we investigate six NLSM datasets, including SNLI, MultiNLI . All classifiers are Random Forests if no specific configuration is mentioned. The classifiers are trained with the training set, and we report the results on the testing set. Predicting semantic relationships without using sentence contents seems impossible. However, we find that the graph-based features (Leakage and Advanced) make the problem feasible on a wide range of datasets. Specifically, on the datasets like QuoraQP and ByteDance, the leakage features are even more effective than the unlexicalized features. One exception is that on MultiNLI, Majority outperforms Leakage and Advanced significantly. Another interesting finding is that on SNLI and ByteDance, advanced graph-based features improve a lot over the leakage features, while on QuoraQP, the difference is very small. Among all the tested datasets, only MSRP and SICKNLI are almost neutral to the leakage features. Note that their sizes are relatively small with only less than 10k samples.