Is Word Segmentation Necessary for Deep Learning of Chinese Representations?
Table 5: Results on the En-Ch machine translation task.
['TestSet', 'Seq2Seq +Attn (word)', 'Seq2Seq +Attn (char)', 'Seq2Seq +Attn+BOW', 'Seq2Seq (char) +Attn+BOW']
As can be seen, the char-based model outperforms the word-based model by a huge margin (+3.13), and this margin is greater than the improvement in the Ch-En translation task. This is because in Ch-En translation, the difference between word-based and char-based models is only present in the source encoding stage, whereas in En-Ch translation it is present in both the source encoding and the target decoding stage. Another major reason that contributes to the inferior performance of the word-based model is the UNK word at decoding time, We also implemented the BPE subword model Sennrich et al. The BPE model achieves a performance of 41.44 for the Seq2Seq+attn setting and 44.35 for bag-of-words, significantly outperforming the word-based model, but still underperforming the char-based model by about 0.8-0.9 in BLEU.