Universal Word Segmentation: Implementation and Interpretation
Table 7: Segmentation accuracies on Arabic and Hebrew with different ways of transducing non-segmental multiword tokens.
['[EMPTY]', 'None', 'Dictionary', 'Transducer', 'Mix']
In the condition None, the identified non-segmental multiword tokens remain unprocessed. In Dictionary, they are mapped via the dictionary derived from training data if found in the dictionary. In Transducer, they are all transduced by the attention-based encoder-decoder. In Mix, in addition to utilising the mapping dictionary, the non-segmental terms not found in the dictionary are transduced with the encoder-decoder. The results show that when the encoder-decoder is applied alone, it is worse than only using the dictionaries, but additional improvements can be obtained by combining both of them.