Revisiting Few-sample BERT Fine-tuning
Table 4: Comparing BERT fine-tuning with mixed precision and full precision. The difference between the two numbers on any dataset is not statistically significant.
['[EMPTY]', 'CoLA', 'MRPC', 'RTE', 'STS-B']
Mixed precision training can accelerate model training while preserving performance by replacing some 32-bit floating-point computation with 16-bit floating-point computation. Transformers uses O1-level optimized mixed precision training implemented with the Apex library. We evaluate if this mixed precision implementation influences our results. We fine-tune BERT with 20 random trials on RTE, MRPC, STS-B, and CoLA. We use two-tailed t-test to test if the distributions of the two methods are statistically different. The performance of mixed precision matches the single precision counterpart, and there is no statistically significant difference.