Neural Machine Translation with 4-Bit Precision and Beyond
Table 1: 4-bit Transformer quantization performance for English to German translation, measured in BLEU score. We explore different method to find the scaling factor, as well as skipping bias quantization and retraining.
['[BOLD] Method', '[BOLD] Scaling  [BOLD] Fixed', '[BOLD] Max', '[BOLD] Optimized', '[BOLD] Fixed', '[BOLD] Max', '[BOLD] Optimized']
Using a simple, albeit unstable max-based scaling has shown to perform better compared to the fixed quantization scale. However, fitting the scaling factor to minimize the quantization squared-error produces the best quality. Interestingly, the BLEU score differences between quantization centers are diminished after retraining.