Is Word Segmentation Necessary for Deep Learning of Chinese Representations?
Table 6: Results on the LCQMC and BQ corpus.
['[BOLD] Dataset', '[BOLD] description', '[BOLD] char valid', '[BOLD] word valid', '[BOLD] char test', '[BOLD] word test']
As can be seen, the char-based model significantly outperforms the word-based model by a huge margin, +1.34 on the LCQMC dataset and +2.90 on the BQ set. For this paraphrase detection task, the model needs to handle the interactions between sub-units of a sentence. We conclude that the char-based model is significantly better in this respect.