Finnish Language Modeling with Deep Transformer Models
TABLE VII: Tr-XL test perplexity scores, (-): The experiment models are not available
['Model', 'Mem-seg len 150-150', '32-32', '(prev best)']
As opposed to BERT, the left+right-marked models have a better training performance than their counterpart. Interestingly the larger model trains much better when compared to providing larger contexts. The same set of parameters for the 32-32 model cannot be replicated for 150-150 model as the latter takes a lot of space on the GPU card. The test performance is similar to that of the training performance with left-right-marked large model(32-32) performing the best.