A Monolingual Approach to Contextualized Word Embeddingsfor Mid-Resource Languages
Table 6: UPOS, UAS and LAS scores for the UDPipe 2.0 baseline reported by (Kondratyuk and Straka, 2019), plus the scores for checkpoints at 1, 3, 5 and 10 epochs for all the ELMoOSCAR and ELMoWikipedia. All scores are test scores. Best ELMoOSCAR scores are shown in bold while best ELMoWikipedia scores are underlined.
['Treebank', 'Model', 'UPOS', 'UAS', 'LAS']
An important topic we wanted to address with our experiments was that of overfitting and the number of epochs one should train the contextualized embeddings for. The ELMo authors have expressed that increasing the number of training epochs is generally better, as they argue that training the ELMo model for longer reduces held-out perplexity and further improves downstream task performance. This is why we intentionally fully pre-trained the ELMoWikipedia to the 10 epochs of the original ELMo paper, as its authors also expressed concern over the possibility of overfitting for smaller corpora. We thus save checkpoints for each of our ELMo model at the 1, 3, 5 and 10 epoch marks so that we can properly probe for overfitting. Thus we believe that talking in terms of training steps as opposed to training epochs might be a more transparent way of comparing two pre-trained models.