On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression
Table 1: The table shows the total parameters, perplexity, and compression efficiency (lower is better) on PTB Language Modeling task. ‡ We reproduced the results.
['[BOLD] PTB', '[BOLD] Param.', '[BOLD] w/o fine-tuning  [BOLD] PPL', '[BOLD] w/o fine-tuning  [BOLD] E(r)', '[BOLD] w/ fine-tuning  [BOLD] PPL', '[BOLD] w/ fine-tuning  [BOLD] E(r)']
In PTB, we fine-tune the compressed model for several epochs. It is clear that compressing Wh works notably better than Wi. We achieve similar results for WT-2. In general, SVD has the lowest perplexity among others. This difference becomes more evident for higher compression (e.g., r=10). Using fine-tuning with rank 10 all the methods we achieve a small improvement compared to the baseline with a 2.13x speedup.