HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
Table 3: Raspberry Pi ARM CPU latency and BLEU comparisons with different models on WMT’14 En-De. HAT has the lowest latency with the highest BLEU.
['[EMPTY]', 'Latency', 'BLEU']
We scale down all models to have similar BLEU scores with Levenshtein for fair comparisons. We adopt the average iteration time of 2.88 for decoding Gu:2019levenshtein, without limiting the length of the output sentence (12 tokens after decoding). HAT runs 1.3× faster than Transformer with higher BLEU; 1.9× faster than Levenshtein with 0.7 higher BLEU. Under similar latency, HAT also outperforms Lite Transformer. These results demonstrate HAT’s effectiveness in lower latency scenarios. Our framework can also be adopted to speedup those models.