Generating Long and Informative Reviews with Aspect-Aware Coarse-to-Fine Decoding
Table 6: Fluency evaluation on Amazon dataset.
['Measures', 'Gold', 'ExpansionNet', 'LeakGAN', 'Our']
Fluency Evaluation. We continue to evaluate the usefulness of the sketch generation module in improving the fluency of the generated text. Following Xu et al. We randomly choose 200 samples from test set. A sample contains the input contexts (i.e., user, item, rating), and the texts generated by different models. It is difficult to develop automatic evaluation methods for accurate fluency evaluation. Here, we invite two human annotators (excluding the authors of this paper) who have good knowledge in the domain of electronic reviews to assign scores to the generated reviews. In the 5-point Likert scale, 5-point means “very satisfying”, while 1-point means “very terrible”. We further average the two annotated scores over the 200 inputs. We can see that our model achieves the highest fluency score among the automatic methods. By using sketches, our model is able to leverage the learned syntactic patterns from available reviews. The Cohen’s kappa coefficients are above 0.7, indicating a high correlation and agreement between the two human annotators.