Decomposable Neural Paraphrase Generation
Table 8: Human Evaluation in WikiAnswers→Quora
['Models', 'Mean Rank', 'Agreement']
To further verify the improvement of Adapted DNPG, we conduct the human evaluation on the WikiAnswers→Quora setting. We have six human assessors to evaluate 120 groups of paraphrase candidates given the input sentences. Each group consists of the output paraphrases from MTL+Copy, DNPG and Adapted DNPG as well as the reference. The evaluators are asked to rank the candidates from 1 (best) to 4 (worst) by their readability, accuracy and surface dissimilarity to the input sentence. The detailed evaluation guide can be found in Adapted DNPG again significantly outperforms MTL+Copy by a large margin (p-value <0.01). The performance of the original DNPG and MTL+Copy has no significant difference (p-value =0.18). All of the inter-annotator agreement is regarded as fair or above.