Extreme Language Model Compression with Optimal Subwords and Shared Projections
Table 2: Masked language modeling task accuracy for the distilled student models and a fine-tune-from-scratch baseline. We observe consistently better performance for our proposed approaches.
['[BOLD] Model ↓ Hidden dimension →', '[BOLD] 48', '[BOLD] 96', '[BOLD] 192']
We observe that dual training significantly improves over the baseline for all model dimensions, and that both shared projection losses added to dual training further improve the word prediction accuracy. It is interesting to note that for all model dimensions, SharedProjUp projecting into the teacher space outperforms SharedProjDown, significantly so for dimension 48. Expectedly, there is a noticeable performance drop going from 192 to 96 to 48-dimensional hidden state models. Note that because of the differing teacher and student model vocabularies, masked word prediction accuracy for the teacher BERTBASE model is not directly comparable with the student models.