Fully Quantized Transformer for Machine Translation
Table 2: Performance of our quantization method on the WMT14 EN-DE and WMT14 EN-FR test set for a fixed number of training steps.
['Model', 'Method', 'Precision', 'EN-DE PPL', 'EN-DE BLEU', 'EN-DE Size (Gb)', 'EN-DE Compr.', 'EN-FR PPL', 'EN-FR BLEU', 'EN-FR Size (Gb)', 'EN-FR Compr.']
We compare our results with two full-precision Transformers: base and big variants. We also evaluate two other quantization approaches. The first one is the ”default” approach, which is to naively quantize every possible operation. In all cases except for post-quantization, BLEU was computed on the test set using the checkpoint which scored the highest accuracy on the validation set. Towards the end of training, we ran one validation epoch for every 100 training steps. Baselines and FullyQT 8-bit results were averaged over 5 trials. Standard deviation of the BLEU scores did not seem higher for any method and ranged between 0.09 and 0.51. Training with quantization was about twice as slow as with the baselines. As for post-training quantization, the BLEU score was computed on the test set using the best validation performance out of 20 trials. The default approach’s nan in the EN-FR task is due to numerical instability. By quantizing every operation, zeros in the LayerNorm ’s denominator are more frequent.