Soft Label Memorization-Generalization for Natural Language Inference
Table 3: Test accuracy results for incorporating SLMG for NLI. Refer to §3.3 for descriptions of the baselines. Highest accuracy result for each model is bolded (one per column).
['[BOLD] Experiment', '[BOLD] Model  [BOLD] LSTM', '[BOLD] Model  [BOLD] NSE', '[BOLD] Model  [BOLD] ESIM']
For each model on the NLI task, we are able to improve generalization performance (i.e. test set accuracy) by injecting soft labeled data at some point. Note that the best performance with SLMG varies according to the model, but for each model there is some configuration that does improve performance. As with all model training, the effect of SLMG requires experimentation according to the use case. In all cases, using CCE as the loss function performs better than using MSE. We suspect that this is due to the fact that small differences are penalized less with CCE than with MSE.