A Comparative Study of Pretrained Language Models on Thai Social Text Categorization
5: Training Loss and Perplexity After Pretraining
['Model', 'Loss', 'Perplexity']
BERT has the lowest word-level cross-entropy loss with 15.3857 MLM perplexity. This is expected due to the difference of the MLM prediction task with fully visible beginning and ending context, providing more contextual information to predict the masked word as compared with traditional forward and backward models. In the domain of traditional autoregressive models, GPT has a lower perplexity than ULMFiT. ELMo is not compared to other models due to prediction granularity difference and is reported as is.