Fully Quantized Transformer for Machine Translation
Table 5: Impact of delaying quantization. Results are on the WMT14 EN-DE and WMT14 EN-FR test set.
['Quantization Start (training step)', 'EN-DE PPL', 'EN-DE BLEU', 'EN-FR PPL', 'EN-FR BLEU']
Our method’s goal is to increase computational efficiency when inferring with the Transformer. To this end, our quantization scheme only requires us to learn s and xmin. Although we do so throughout the whole training, this is not a necessity. Quantization could also be applied later during training. The earliest we start quantizing is at 100 steps, since we need at least a few steps to assess the running estimates. All models were evaluated on the WMT14 EN-DE and WMT14 EN-FR translation tasks. BLEU was measured on the test set using the checkpoint which scored the highest accuracy on the validation set during training. Validation was computed every 100 training steps towards the end of training. From our observed results, quantizing the model early on seems preferable.