BERTje: A Dutch BERT Model
Table 2: Part-of-speech tagging accuracy scores for Lassy Small and SoNaR.
['Model', 'UD-LassySmall Train', 'UD-LassySmall Dev', 'UD-LassySmall Test', 'SoNaR-1 (coarse) Train', 'SoNaR-1 (coarse) Dev', 'SoNaR-1 (coarse) Test', 'SoNaR-1 (fine-grained) Train', 'SoNaR-1 (fine-grained) Dev', 'SoNaR-1 (fine-grained) Test']
BERTje does outperform multilingual BERT consistently, but the 850K checkpoint model appears to perform just as well as the fully trained BERTje model. For all three tag sets, the difference between the 850K checkpoint and the fully pre-trained BERTje model is at most 0.3 percentage points. This indicates that the model has already learned the relevant information before the 850K checkpoint. This is important to acknowledge since the previously mentioned NER results shows that the model does learn new information that is relevant for named-entity recognition after this checkpoint.