PMIndia – A Collection of Parallel Corpora of Languages of India
Table 5: Bleu scores for NMT systems built on the PMIndia corpus, for English to/from languages of India, setting aside 1000 sentences for dev and 1000 for test.
['Language', 'en-X', 'X-en']
We evaluated on tokenized text, using the multi-bleu.perl script from Moses, since sacrebleu post1:2018:WMT does not currently support South Asian languages. we observer very low bleu scores for translation into languages of India, especially for the Dravidian languages (Kannada, Malayalam, Tamil and Telugu). The Dravidian languages are all agglutinative, making inward translation challenging, but also making evaluation of translation using word-based metrics like bleu less reliable. For the languages with very small (less than 10k) data set sizes , the translation from/to Assamese shows poor results, as expected, but translation into and out of Manipuri and Urdu scores much higher, despite the data set size. It’s possible that the translation sample for these languages is more domain-specific than for other languages. For translation into English, the scores are nearly always higher, and sometimes much higher. This observation suggests that the low scores for translation into languages of India are mainly caused by the difficulties posed by the languages themselves, and not by data set problems.