D-PAGE: Diverse Paraphrase Generation
(b) Syn-Scale
['Dec Id', 'References r1', 'References r2', 'References r3', 'References r4', 'References r5']
In order to understand the capability of capturing specific rewriting patterns, we conduct experiments on synthetic datasets for our model. In particular, we apply K decoders to generate K sets of paraphrases. Since we have a reference set per rewriting pattern, we pair each paraphrase set with each reference set. For each pair, we align each system output to a single reference, so that we are able to compute a BLEU score for that pair. If a model is able to capture all patterns, we expect that for each pattern, there is one decoder achieving a high score. For Syn-Sub, our model is able to learn the patterns r1, r3 and r4, but only partially capture the pattern r2. Our further investigation shows that D5 tends to provide outputs with mixed rewriting pattern between r2 and r4 . For Syn-Scale, although the lengths of input and output sequences are different and the dataset include two types of operations, substitution and insertion, our model can still learn all possible patterns perfectly. We contribute this to smaller vocabulary size and shorter sequence length. There are mainly two types of behavior for both Noise-K and VAE-K, on our synthetic datasets. i) The models totally get lost among different rewriting patterns, and fail to generate coherent results, such as Noise-K on Syn-Sub and Syn-Scale and VAE-K on Syn-Sub. ii) The models manage to learn one of the rewriting patterns, while they ignore other possible ways to paraphrase, such as VAE-K on Syn-Scale. Comparing with the results of D-PAGE, in