Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation
Table 4: Average WikiAnn NER F1 scores on the development sets of 265 languages with shared vocabularies of different size.
['[EMPTY]', 'BPE vocabulary size 100k', 'BPE vocabulary size 320k', 'BPE vocabulary size 1000k']
The shared BPE vocabulary and corresponding byte-pair embeddings allow training a single NER model for all 265 languages. To do so, we first encode WikiAnn in all languages using the shared BPE vocabulary and then train a single multilingual NER model in the same fashion as a monolingual model. As the vocabulary size has a large effect on the distribution of BPE symbol lengths To reduce the disparity between dataset sizes of different languages, and to keep training time short, we limit training data to a maximum of 3000 instances per language. without further finetuning -finetune show a strong improvement in low-resource languages (89.7 vs. 81.9 with +someshape), while performance degrades drastically on high-resource languages. In multilingual training, certain aspects of the task at hand, such as tag distribution and BIO constraints have to be learned only once, while they have to be separately learned on each language in monolingual training. Furthermore, multilingual training may prevent overfitting to biases in small monolingual datasets, such as a skewed tag distributions. Before training, distinct clusters of subword embeddings from the same language are visible. After training, some of these clusters are more spread out and show more overlap, which indicates that some embeddings from different languages appear to have moved “closer together”, as one would expect embeddings of semantically-related words to do. However, the overall structure of the embedding space remains largely unchanged. The model maintains language-specific subspaces and does not appear to create an interlingual semantic space which could facilitate cross-lingual transfer.