Universal Word Segmentation: Implementation and Interpretation
Table 14: Comparison between the universal model and the language-specific models.
['[EMPTY]', 'This Paper', 'Shao', 'Che', 'Björkelund']
With pretrained character embeddings, ensemble decoding and joint POS tags prediction as introduced in \newciteshao17, considerable improvements over the universal model presented in this paper can be obtained. However, the joint POS tagging system is difficult to generalise as single characters in space-delimited languages are usually not informative for POS tagging. Additionally, compared to Chinese, sentences in space-delimited languages have a much greater number of characters on average. Combining the POS tags with segmentation tags drastically enlarges the search space and therefore the model becomes extremely inefficient both for training and tagging. The joint POS tagging model is nonetheless applicable to Japanese and Vietnamese. \newciteche-EtAl:2017: K17-3 build a similar Bi-LSTM word segmentation model targeting languages without space delimiters like Chinese and Japanese. The proposed model incorporates rich statistics-based features gathered from large-scale unlabelled data, such as character unigram embeddings, character bigram embeddings and the point-wise mutual information of adjacent characters. \newcitebjorkelund-EtAl:2017: K17-3 use a CRF-based tagger for multiword token rich languages like Arabic and Hebrew. A predicted Levenshtein edit script is employed to transform the multiword tokens into their components. The evaluation scores on a selected set of languages reported in \newciteche-EtAl:2017: K17-3 and \newcitebjorkelund-EtAl:2017: