Finnish Language Modeling with Deep Transformer Models
TABLE III: BERT training performance
['Model', 'Masked LM Loss', 'Masked LM Accuracy']
This set of parameters were chosen as there training performances were better than smaller models on modelling the long sequences of sub-words. A maximum sequence length of 300 encompasses 98 percent of the training data and also allows us to fit larger models on the GPU card. Hyper-parameter optimization is very difficult in case of these models as they take around 15 days to train given the resources. The hyperparameter choices were therefore more dependant on the original BERT with little tweaks.