A Comparative Study of Pretrained Language Models on Thai Social Text Categorization
8: Limited Pretraining Corpus Results. The public and private scores are calculated with approximately 30% and 70% of the test data respectively.
['Model', 'Wisesight (Acc.) Private', 'Wisesight (Acc.) Public', 'Wongnai (Acc.) Private', 'Wongnai (Acc.) Public']
We also investigated the performance of the models in the scenario where the pretraining corpus is limited. This result reflects the learning ability of the models in a language where training data is scarce. We randomly sampled a total of 40M tokens (equivalent to around 234K threads) from the dataset used in our previous experiments. ULMFiT, ELMo, and GPT are trained for 3 epochs while BERT is trained for 30k steps (equivalent to approximately 6.5 epochs on this data). On the other hand, adding ELMo to LSTM input shows little improvement. This means that ELMo requires a larger corpus to be effective. Although BERT performs well on the Wisesight dataset, it has a drop in performance on Wongnai dataset.