Basic Linguistic Resources and Baselines for Bhojpuri, Magahi and Maithili for Natural Language Processing
Table 10. Perplexity of Purvanchal Languages and Hindi (Here * indicates the PP over the equal number of sentences according to Maithili corpus)
['[BOLD] PP', '[BOLD] Unigram', '[BOLD] Bigram', '[BOLD] Trigram', '[BOLD] Unigram*', '[BOLD] Bigram*', '[BOLD] Trigram*']
We also notice that the values for minimum corpus size seem to be more meaningful for perplexity comparison than full corpus size, which intuitively makes sense, since the dataset size has a large effect on the perplexity values. Therefore, we focus on the equal corpus size values in this table. From these, we can observe that the value of perplexity for Bhojpuri becomes less than half when we go from unigram to bigram, but decreases only slightly when we go further to trigrams. This may be interpreted to mean that two word word groups (or chunks) are more syntactically relevant for Bhojpuri then tree word groups. In case of Magahi, the value drops steeply from unigrams to bigrams and then doesn’t decrease so much. This perhaps means that even two word groups are not very relevant for Magahi from syntactic point of view. This statement has to be qualified by the fact that n-gram models can model syntax only to a limited extend, one of the reasons being that they cannot model long distance dependencies. Therefore, it indicates the possibility that Magahi has much more of long distance dependencies. The same pattern as Magahi is evident in Maithili, but to an order of magnitude greater. Maithili, thus relies the most on morphology and long distance dependencies than the other three languages. The trend for Hindi very similar to Bhojpuri, which matches the linguistic intuition that these two languages have the highest similarity, at least as far as is visible in written corpus available to us.