HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
Table 8: Specific latency numbers, BLEU and SacreBLEU scores for searched HAT models in Figure 7 and Figure 8.
['Task', 'Hardware', 'Latency', 'BLEU', 'SacreBLEU']
Inference latency and BLEU trade-offs of WMT’14 En-De and En-Fr on three hardware platforms. HAT consistently outperforms the baseline Transformers and achieves up to 3× faster inference speed and 3.7× smaller size over Transformer-Big. The embedding dims are 512 and 1024 for the Transformer-Base and Big, respectively. The hidden dims are 4× and 2× of the embedding dim for WMT and IWSLT. The IWSLT models are smaller to prevent overfitting Wu:2019payless. We obtain a series of baseline models with layer number scaling (yellow) and dimension scaling (blue). We set different latency constraints on three hardware to get a series of HAT models. HAT consistently outperforms baselines with a large gap under different latency constraints. On ARM CPU, HAT is 3× faster and 3.7× smaller than Transformer-Big with the same BLEU. On Intel CPU, HAT achieves over 2× speedup. On Nvidia GPU, the blue dash line is nearly vertical, indicating that dimension scaling can hardly reduce the latency. In this case, HAT can still find models with low latency and high performance.