1 Introduction
Table 2: Test Perplexity of DKP compressed medium LM for various training techniques discussed in Section 3.2 and 3.3. CMR techniques leads to the best accuracy for 100× compression factor.
['Compression Factor', 'Training Method', 'Sparsity of  [ITALIC] Msp', 'Test Perplexity']
In BCD we alternate between, only training Mkp, blocking gradient flow to Msp, or train Mkp, blocking gradient flow to Msp. when trained using BCD. However, there is still a small increase in training error with increased sparsity. This can indicate that CMA may have not been completely managed. As you can see, these techniques help us bring the perplexity down to 100.37 from 150.737 originally. However, by reducing the compression factor from 338× to 100×, we are improving the perplexity by approximately 4 points only. As the sparsity of Msp increases, the need for CMR decreases and can be removed entirely. The training perplexity during the training phase does not increase as the sparsity of the Msp matrix increases.