Differentially Private Distributed Learning for Language Modeling Tasks
Table 1: Random rehearsal vs learning without forgetting. For LwF mode λ is a coefficient of the ground truth probability distribution in the loss function (1)-(2). For random rehearsal mode λ is a portion of user training data in on-device training.
['Method', 'Standard English dataset (Wikipedia) PPL', 'Standard English dataset (Wikipedia) KSS, %', 'User dataset (Twitter) PPL', 'User dataset (Twitter) KSS, %', 'Av. PPL']
We see that the performance gap between the standard English and the user test sets can be considerably reduced at the cost of performance degradation on the first dataset. The best average perplexity is reached with the random rehearsal method and λ=0.5. We believe that the reason of the comparably inferior performance of the LwF method can be explained by the fact that soft labels used by LwF give a poor approximation of the true word distribution of general English so adding a small portion of true data gives better results in terms of knowledge preservation.