Differentially Private Distributed Learning for Language Modeling Tasks
Table 2: Averaging vs transfer learning for server-side model update.
['Method', 'Standard English dataset (Wikipedia) PPL', 'Standard English dataset (Wikipedia) KSS, %', 'User dataset (Twitter) PPL', 'User dataset (Twitter) KSS, %', 'Av. PPL']
We saw no significant differences between transfer learning on real and generated data. The difference between transfer learning and averaging is more sound but still not large. At the same time model averaging is much more computationally efficient, as long as transfer learning requires calculation of labels from each of the teacher models. After 300 rounds of model updates with 3000 nodes (10 nodes per round) we ended up with an 8.7 absolute gain in KSS on the user data test with only a 0.6 absolute KSS drop on the standard English data test.