Self-Knowledge Distillation in Natural Language Processing
Table 2: NLLs for LM with different models on PTB and Wiki-2.
['Model', 'PTB', 'Wiki-2']
The evaluation metric is the negative log-likelihood (NLL) for each sentence (the lower is the better). It shows that our proposed methods (both noise injection and self-distillation knowledge) improve the results in the LM task. Note that SKD provides more knowledgeable information than Gaussian noise.