Joint Copying and Restricted Generation for Paraphrase
Table 3: Performance of different models. ∗Moses simply ignore the unknown words.
['Data', 'Model', 'Informativeness ROUGE-1(%)', 'Informativeness ROUGE-2(%)', 'Text Quality PPL', 'Text Quality Length', 'Text Quality UNK(%)', 'Text Quality Copy(%)']
In this table, the metrics that measure informativeness and text quality are separated. Let’s look at the informativeness performance first. As can be seen, CoRe achieves the highest ROUGE scores on both summarization and text simplification. In contrast, the standard attentive Seq2Seq model ABS is slightly inferior to Moses. It even performs worse than the simple baseline LEAD in terms of ROUGE-2 in summarization. Apparently, introducing the copying and restricted generation mechanisms is critical for the paraphrase-oriented tasks. Then, we check the quality of the generated sentences. According to PPL, the sentences produced by CoRe resemble the target language the most. It is interesting that LEAD extracts human-written text in the source. Nevertheless, its PPL is considerably higher than CoRe on both datasets. It seems that CoRe indeed captures some characteristics of the target language, such as the diction. We also find that the PPL of Moses is the largest, and its generated length reaches the length of the source text. Moses seems to conduct word-to-word translation. This practice is acceptable in text simplification, but totally offends the summarization requirement. Although not manually controlled, the lengths of the outputs in ABS and CoRe are both similar to the actual one, which demonstrates the learning ability of Seq2Seq models. The former verifies the power of our two decoders, while the latter may be attributed to the supplement of the supervision of writing modes.