Finnish Language Modeling with Deep Transformer Models
TABLE VI: Tr-XL training perplexity scores
['Model', 'Mem-seg len', '[EMPTY]']
From the above parameter choice, we wanted to experiment, whether providing more Segment and Memory length is advantageous (longer context) than a larger model. These parameters where chosen after some hyperparameter optimization.