Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
Table 8: AdamW stops scaling at the batch size of 16K. The target F1 score is 90.5. Lamb achieves a F1 score of 91.345. The table shows the tuning information of AdamW. In this table, we report the best F1 score we observed from our experiments.
['Solver', 'batch size', 'warmup steps', 'LR', 'last step infomation', 'F1 score on dev set']
AdamW stops scaling at the batch size of 16K. The target F1 score is 90.5. Lamb achieves a F1 score of 91.345.