TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing
Table 3: Model sizes of teacher and students. The number of parameters includes embeddings but does not include output layers.
['[BOLD] Model', '[BOLD] # Layers', '[BOLD] Hidden size', '[BOLD] Feed-forward size', '[BOLD] # Parameters', '[BOLD] Relative size']
Models. All the teachers are BERT\tt BASE-based models. We test the performance of the following student models: T6 and T3 are BERT\tt BASE with fewer layers of transformers. Especially, T6 has the same structure as DistilBERT Sanh et al. T3-small is a 3-layer BERT with half BERT-base’s hidden size and feed-forward size. T4-tiny is the same as TinyBERT, a 4-layer model with an even smaller hidden size and feed-forward size. BiGRU is a single-layer bidirectional GRU. Its word embeddings are taken from BERT\tt BASE. T3-small and T4-tiny are initialized randomly.