Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation
Table 2: Hyper-parameters of various model sizes used in this work. For example, the Large Transformer model architecture uses 6 encoder and 6 decoder layers, a feed-forward hidden dimension of 2048 at each layer, and a word-embedding / hidden dimension of 512.
['Size', 'Layers', 'FF Size', 'Hidden Size']
Teachers are trained using the Large hyper-parameter settings, while we experiment with Medium, Small, and Tiny students for each configuration and language/domain setting. All models are Transformers [Vaswani2017-vu]. Models are trained either for 300,000 updates, 100 epochs, or until the model does not improve for 10 checkpoints (early-stopping), whichever comes first.