Joint Copying and Restricted Generation for Paraphrase
Table 2: Target word coverage ratio (%) on the test set.
['Vocabulary', 'Summarization', 'Simplification']
In this paper, we develop a novel Seq2Seq model called CoRe, which captures the two core writing modes in paraphrase, i.e., Copying and Rewriting. CoRe fuses a copying decoder and a restricted generative decoder. Therefore, the weights learned by the attention mechanism have the explicit meanings in the copying mode. Meanwhile, the generative decoder produces the words restricted in the source-specific vocabulary. This vocabulary is composed of a source-target word alignment table and a small set of frequent words. The alignment table is trained in advance, and many frequent rewriting patterns are included in it. It seems better to update the alignment table according to the learned attention weights. However, with the supplement of a few frequent words, experiments (see Table: While the output dimension of our generative decoder is just one tenth of the output dimension used by the common Seq2Seq models, it is able to generate highly relevant words concerning rewriting. To combine the two decoders and determine the final output, we develop a predictor to predict the writing mode of copying or rewriting. Since we know the actual mode at each output position in a training instance, we introduce a binary sequence labeling task to guide the learning of this predictor, which takes advantages of the supervision derived from the writing modes. It appears that both datasets hold a high copying ratio. When we restrict the generative decoder to produce the source alignments, more than 85% target words can be covered. When combined with 2000 frequent words, the coverage ratio of our model is already close to that using the vocabulary of 30000 words. We conduct text quality evaluation from several points of view. The lower PPL usually means higher readability. We also perform the statistical analysis on the average length of the target text, UNK ratio and copy ratio.