Parsimonious Morpheme Segmentation with an Application to Enriching Word Embeddings
TABLE IV: Language modeling task
['[BOLD] Dataset', '[BOLD] English', '[BOLD] German', '[BOLD] Turkish']
We observe that across all segmentation-based morpheme-enriched embeddings perform better in language modeling over traditional skip-gram. In contrast, FastText’s indiscriminate enumeration of all possible morphemes appears to perform much poorer in this task. Finally, MorphMine outperforms the other morpheme enriched baselines. This may be due to MorphMine utilizing morphemes of mutliple granularity which closely capture semantic meaning of rare and OOV words at the largest granularity.