Morphological and Language-Agnostic Word Segmentation for NMT This work has been supported by the grants 18-24210S of the Czech Science Foundation, SVV 260 453 and “Progress” Q18+Q48 of Charles University, H2020-ICT-2014-1-645452 (QT21) of the EU, and using language resources distributed by the LINDAT/CLARIN project of the Ministry of Education, Youth and Sports of the Czech Republic (projects LM2015071 and OP VVV VI CZ.02.1.01/0.0/0.0/16 013/0001781). We thank Jaroslava Hlaváčová for digitizing excerpts of [slavickova1975] used as gold-standard data for evaluating the segmentation methods.
Table 2: Data characteristics and automatic metrics after 300k steps of training.
['[BOLD] de', '[BOLD] cs', '[BOLD] tokens  [BOLD] de', '[BOLD] tokens  [BOLD] cs', '[BOLD] types  [BOLD] de', '[BOLD] types  [BOLD] cs', '[BOLD] %  [BOLD] shrd', '[BOLD] BLEU', '[BOLD] CharacTER', '[BOLD] chrF3', '[BOLD] BEER']
Since the vocabulary size after Morfessor or DeriNet splitting alone often remains too high, we further split the corpus with BPE or STE. Unfortunately, none of the setups performs better than the STE baseline.