Soft Label Memorization-Generalization for Natural Language Inference
Table 4: Confusion matrices for the LSTM model, trained according to the baseline (first block), using SLMG-S with CCE (second block), and using SLMG-S with MSE (third block). Gold standard labels run down the left hand side, while predicted labels are across the top in the matrix. The highest count of True Positives for each label across the three model-training setups are bolded.
['[EMPTY]', 'E', 'E 2739', 'C 191', 'N 438']
To better understand the effects of SLMG on generalization, we look at the changes in test set performance when SLMG is used as compared to the baseline case. In both cases of training with SLMG, the number of correctly classified entailment and contradiction examples increased, while the number of neutral examples correctly classified decreased. However when MSE is used as the soft label loss function, the increase in misclassified neutral examples was enough to offset the gains in correctly classified entailment and contradiction examples. Depending on the use case, this result could be useful for applications. Fewer false negatives for entailment and contradiction examples may be more important than fewer true positives for the neutral class. In fact, the SLMG-MSE method outperforms SLMG-CCE in the binary task (88.0% vs. 86.6%) due to the fact that its performance on the entailment label is much higher.