HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
Table 5: The performance of SubTransformers with inherited weights are close to those trained from-scratch, and have the same relative performance order.
['WMT’14 En-De Inherited Val Loss', 'WMT’14 En-De Inherited BLEU', 'WMT’14 En-De From- Scratch BLEU', 'WMT’14 En-Fr Inherited Val Loss', 'WMT’14 En-Fr Inherited BLEU', 'WMT’14 En-Fr From- Scratch BLEU']
SubTransformer Performance Proxy. All SubTransformers inside the SuperTransformer are uniformly sampled and thus equally trained, so the performance order is well-preserved during training. The BLEUs of SubTransformers with inherited weights and weights trained from-scratch are very close. More importantly, they also have the same relative performance order. Therefore, we can rely on the proxy to search high-performance model architecture, significantly reducing the search cost.