Basic Linguistic Resources and Baselines for Bhojpuri, Magahi and Maithili for Natural Language Processing
Table 12. Cross-lingual similarity score induced after applying character level language model using SSNGLMScore
['[BOLD] Language Model', '[BOLD] Bhojpuri', '[BOLD] Test Language  [BOLD] Maithili', '[BOLD] Magahi', '[BOLD] Hindi']
These are symmetric scores, as opposed to the cross entropy scores. Bhojpuri is closest to Magahi and almost equally distant from Hindi and Maithili. Maithili is the most distant from Hindi, but almost equally close to the other two Purvanchal languages. Magahi is closest to Maithili, then to Bhojpuri and the least to Hindi. This more or less matches the linguistic knowledge, but we do not see associativity in terms of language similarity, at least using this measure of similarity. Some of this may be due chance, because of taking random samples of 50% corpus size for training and testing. Perhaps n-fold cross validation method will give us more reliable estimates.