Extreme Language Model Compression with Optimal Subwords and Shared Projections
Table 3: Results of the distilled models, the teacher model and baselines on the downstream language understanding task test sets, obtained from the GLUE server, along with the size parameters and compression ratios of the respective models compared to the teacher BERTBASE. MNLI-m and MNLI-mm refer to the genre-matched and genre-mismatched test sets for MNLI.
['[BOLD] Model', '[BOLD] Hidden', '[BOLD] Vocab', '[BOLD] Compress', '[BOLD] MRPC', '[BOLD] MNLI-m', '[BOLD] MNLI-mm', '[BOLD] SST-2']
We note that models trained with our proposed approaches perform strongly and consistently improve upon the identically parametrized NoKD baselines, indicating that the dual training and shared projection techniques are effective, without incurring significant losses against the BERTBASE teacher model. Comparing with the PKD baseline, our 192-dimensional models, achieving a higher compression rate than either of the PKD models, perform better than the 3-layer PKD baseline and are competitive with the larger 6-layer baseline on task accuracy while being nearly 5 times as small.