An Investigation of Transfer Learning-Based Sentiment Analysis in Japanese
Table 7: Results from different parameter updating strategies. BCN+ELMo and BERTBASE are original implementations. BCN+ELMo unfreeze shows experimental results of fine-tuning both BCN and ELMo layers on target dataset while BERTBASE freeze is where BERTBASE layers are frozen and only classifier layer fine-tuned on target dataset
['Model', 'Yahoo Binary']
In its original implementation, when BERT is fine-tuned, all of its layers are trained. This is quite different from fine-tuning ELMo, where its layers are frozen and only task-specific models ( BCN in our case) are updated. The hidden state associated to the first character of the input is pooled and provided to a linear layer that sits on top. This way, BERT is computationally much cheaper and faster. Result shows that using BERT as a feature extractor shows competitive performance. Unfreezing ELMo Pre-trained ELMo weights are used for initialization as well; however, weights are changed along with BCN layers. This experiment allows us to compare the performance of freezing/unfreezing ELMo layers.