Improving Lemmatization of Non-Standard Languages with Joint Learning
Table 3: Average accuracy across morphologically related standard languages. Type 1 encloses bg, cs, lv, ru and sl. Type 2 comprises et, fi, hu, tr. Finally, Type 3 encompasses de, en, es, fr, it and nb.
['[EMPTY]', 'Full Type 1', 'Full Type 2', 'Full Type 3', 'Ambiguous Type 1', 'Ambiguous Type 2', 'Ambiguous Type 3', 'Unknown Type 1', 'Unknown Type 2', 'Unknown Type 3']
It is apparent that the Edit-tree approach is very effective for Type 3 languages both in ambiguous and unknown tokens. In both Type 1 and Type 2 languages, the best overall performing model is Sent-LM. In the case of ambiguous tokens, Sent-LM achieves highest accuracy for Type 1 languages, but it is surpassed by the Edit-tree approach on Type 2 languages. Finally, in the case of unknown tokens, we observe a similar pattern to the historical languages where Plain and Sent have an advantage over Sent-LM.