Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
Table 2: Effects of group-wise quantization for Q-BERT on three tasks. The quantization bits were set to be 4 for weights, 8 for embeddings and 8 for activations on all the tasks. From top to down, we increase the number of groups. In order to balance the accuracy and hardware efficiency, we set 128 groups for other experiments.
['# Group', 'SST-2', 'MNLI-m/mm', 'CoNLL-03']
We measure the performance gains with different group numbers in Tab. We can observe from the table that performing layer-wise quantization (shown in Fig. However, the performance significantly increases as we increase the number of groups. For example, in the case with 12 groups, the performance degradation is less than 1% for all the tasks. Further increasing the group number from 12 to 128 increases the accuracy by at least 0.3% accuracy. However, increasing the group number further from 128 to 768 can only increase the performance within 0.1%. This shows that the performance gain saturates with group number 128, and we should note that the saturation group number can potentially be even smaller, such as 12, with less aggressive quantization. It is also preferable not to have very large value for the number of group since it increases the number of Look-up Tables (LUTs) necessary for each matrix multiplication. This can adversely affect hardware performance, and based on our results there are diminishing returns in terms of accuracy. To ensure continuity, we used 128 groups for both Q-BERT and Q-BERTMP in all our experiments in Sec.