Is Word Segmentation Necessary for Deep Learning of Chinese Representations?
Table 3: Language modeling perplexities in different models.
['model', 'dimension', 'ppl']
In both settings, the char-based model significantly outperforms the word-based model. In addition to Jieba, we also used the Stanford CWS package Monroe et al. Che et al. As can be seen, the char-based model not only outperforms the word-based model, but also the hybrid (word+char) model by a large margin. The hybrid (word+char) model outperforms the word-based model. This means that characters already encode all the semantic information needed and adding word embeddings would backfire. The hybrid (char only) model performs similarly to the char-based model, suggesting that word segmentation does not provide any additional information. It outperforms the word-based model, which can be explained by that the hybrid (char only) model computes word representations only based on characters, and thus do not suffer from the data sparsity issue, OOV issue and the overfitting issue of the word-based model.