Paraphrase Detection on Noisy Subtitles in Six Languages
Table 8: Vocabulary sizes for 2 million phrase pairs from Opusparcus. The values contain both positive and negative examples used in training.
['[EMPTY]', 'No segmentation', 'Morfessor']
The training data sets each consist of two million paraphrase pairs. The Morfessor Baseline algorithm is utilized to split words into smaller subword units, partly resembling morphemes. The segmentation of words into such subword units is clearly effective in reducing the vocabulary size for all languages. As expected, particularly drastic reductions can be seen for Finnish and Russian as they are the most morphologically complex languages of the six.