Neural Polysynthetic Language Modelling
Table 5.10: Word-level perplexity for setting 2 (New Testament only). V denotes vocabulary size for BPE operation
['Language', 'Morfessor', 'BPE (V=500)', 'BPE (V=5k)', 'Character']
After comparing different tokenization methods per language, we compared different languages to see which language is easier or harder to model. This line of inquiry has been pursued by several recent studies (cotterell-etal-2018-languages; mielke-etal-2019-kind; gerz-etal-2018-relation), where various languages are modeled using a state-of-the-art neural language model to compare relative difficulty of modelling a language with particular linguistic features. It should be noted that our data per language were not parallel so the comparison has to be drawn with caution. However, we still attempted the comparison here as comparing our models may provide insights for future studies given that we used the same or very similar RNN language models as the previous literature and that polysynthetic languages have not been discussed in this line of inquiry. all data setting and eng under the 2) New Testament setting. However, character-level perplexity may not be the right metric to use to compare different languages. The problem with the character-level measure is that it does not tell us much about real-life applications, where the difficulty of predicting an entire word might be more meaningful. More importantly, the character-level perplexity underestimates the difficulty of modelling polysynthetic languages as they tend to have longer, morphologically complex words. In fact, when we look at the word-level perplexity, the differences between polysynthetic languages and others become clearer. When considering the difficulty of predicting the next word in the languages than the next character, iku is no longer the easiest to model under any condition. The word-level measure clearly shows that eng, followed by spa, was the easiest to model. Comparisons of the word-level perplexity values suggest that ess, esu and iku are quite similarly hard to model while grn is less difficult even though it is still quite harder to model than language like eng and spa. This observation agrees with our previous observation about the descriptive statistics of the datasets.