HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
Table 6: The SubTransformer inherited from the SuperTransformer can achieve similar or better performance than the same SubTransformer trained from-scratch. Training steps are saved by 4×.
['Task', 'From-Scratch 40K', 'Inherit-Finetune 10K']
Finetuning Inherited SubTransformers In practice, we can also directly finetune the SubTransformers with the inherited weights from the SuperTransformer to further reduce the training cost. With 10K finetuning steps (1/4 of from-scratch training), the inherited SubTransformers can achieve similar or better performance than trained from-scratch ones In this way, the training cost for a model under a new hardware constraint can be further reduced by 4×, since the SuperTransformer training cost is amortizable among all searched models.