Neural Syntactic Preordering for Controlled Paraphrase Generation
Table 4: Human annotated quality across different models. The evaluation was done on a 3 point quality scale, 2 = grammatical paraphrase, 1 = ungrammatical paraphrase, 0 = not a paraphrase.
['Model', '[BOLD] 2', '[BOLD] 1', '[BOLD] 0']
We also performed human evaluation on Amazon Mechanical Turk to evaluate the quality of the generated paraphrases. We randomly sampled 100 sentences from the development set. For each of these sentences, we obtained 3 generated paraphrases from each of the following models: i) SCPN, ii) vanilla sequence-to-sequence and iii) our proposed Sow-Reap model. We follow earlier work kok2010hitting; iyyer2018adversarial and obtain quality annotations on a 3 point scale: 0 denotes not a paraphrase, 1 denotes that the input sentence and the generated sentence are paraphrases, but the generated sentence might contain grammatical errors, 2 indicates that the input and the candidate are paraphrases. To emulate the human evaluation design in iyyer2018adversarial, we sample paraphrases after filtering using the criterion outlined in the previous section and obtain three judgements per sentence and its 9 paraphrase candidates. As we can see, the results indicate that the quality of the paraphrases generated from our model is substantially better than the SCPN model. Furthermore, similar to quantitative evaluation, the human evaluation also demonstrates that the performance of this model is similar to that of the vanilla sequence-to-sequence model, indicating that the inclusion of target rearrangements do not hurt performance.