How to Fine-Tune BERT for Text Classification?
Table 4: Decreasing layer-wise layer rate.
['Learning rate', 'Decay factor  [ITALIC] ξ', 'Test error rates(%)']
5.3.4 on IMDb dataset. We find that assign a lower learning rate to the lower layer is effective to fine-tuning BERT, and an appropriate setting is ξ=0.95 and lr=2.0e-5.