Context-aware Natural Language Generation with Recurrent Neural Networks
Table 4: Results of fake review classification with human judgments (Positive: real reviews, Negative: fake reviews). In all the domains, more than 50% of the fake reviews are misclassified.
['Domain', 'TP', 'FN', 'TN', 'FP']
Human Evaluation. We use the Amazon Mechanical Turk to evaluate whether the reviews are fake not. We divide all the data into different batches. Each batch contains twenty reviews about the same product. We show the urls of the products, the sentiment rating and the review content to users to ask the turkers to judge whether the reviews are written by real users or not. To control the quality of the results, some “gotcha” questions are inserted in the middle of the list of reviews. Only the results judged by users who answer the “gotcha” questions correctly are kept. The kappa score is ….