Fully Quantized Transformer for Machine Translation
Table 3: Effect of quantizing single activations of the Transformer. Results are on the WMT14 EN-FR test set.
['Module', 'Quantized Activation', 'No Bucketing PPL', 'No Bucketing BLEU', 'Bucketing PPL', 'Bucketing BLEU']
To better understand which operations are more sensitive to quantization, we evaluate such effect on single operations of the Transformer. By this, we mean quantizing the operation of a module for all Transformer layers. The effect of bucketing was also evaluated. BLEU was computed on the test set after 100k steps of training. In 24 out of 27 experiments, performance was better than our full-precision baseline of 38.34 BLEU. Solely quantizing the LayerNorm’s denominator with no bucketing results in poor performance. The latter also cannot be bucketed since all dimensions of the variance tensor vary per batch. To successfully quantize this element without causing any loss in performance, we suspect quantizing other elements in the network helps.