ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations
Table 3: Manual evaluation of 100-pair data samples drawn from five ranges of the automatic paraphrase score (first column). Second column shows total count of pairs in that range in ParaNMT-50M. Paraphrase strength and fluency were judged on a 1-3 scale and the table shows counts of each score designation.
['Para.\xa0Score Range', '# (M)', 'Tri. Overlap Mean (Std.)', 'Paraphrase 1', 'Paraphrase 2', 'Paraphrase 3', 'Fluency 1', 'Fluency 2', 'Fluency 3']
For each score range, we report the number of pairs, the mean trigram overlap score, and the number of times each paraphrase/fluency label was present in the sample of 100 pairs. There is noise in the dataset but it is largely confined in the bottom two ranges which together comprise only 16% of the entire dataset. In the highest paraphrase score range, 86% of the pairs possess a strong paraphrase relationship. The annotations suggest that ParaNMT-50M contains approximately 30 million strong paraphrase pairs, and that the paraphrase score is a good indicator of quality. With regards to fluency, the vast majority of the back-translations are fluent, even at the low end of the paraphrase score range. At the low ranges, we inspected the data and found there to be many errors in the sentence alignment in the original bitext.