FinBERT: Financial Sentiment Analysis with Pre-trained Language Models
Table 7. Performance on starting training from different layers
['First layer unfreezed', 'Loss', 'Accuracy', 'Training time']
Fine-tuning only the classification layer does not achieve close performance to fine-tuning other layers. However fine-tuning only the last layer handily outperforms the state-of-the-art machine learning methods like HSC. After Layer-9, the performance becomes virtually the same, only to be outperformed by fine-tuning the whole model. This result shows that in order to utilize BERT, an expensive training of the whole model is not mandatory. A fair trade-off can be made for much less training time with a small decrease in model performance.