Credible Review Detection with Limited Information using Consistency Features
Table 3: Credible review classification accuracy with 10-fold cross validation. TripAdvisor dataset contains only review texts and no user/activity information.
['[BOLD] Models', '[BOLD] Features', '[BOLD] TripAdvisor', '[BOLD] Yelp∗']
(1) Credible review classification : We study the performance of the various approaches in distinguishing a credible review from a non-credible one. Since this forms a binary classification task, we consider a balanced dataset containing equal proportion of data from each of the two classes. On the Yelp dataset, for each item we randomly sample an equal number of credible and non-credible reviews (to obtain Yelp∗); while the TripAdvisor dataset is already balanced. Since the TripAdvisor dataset has only review text, the user/activity models could not be used there. The experiment could also not be performed on Amazon, as the ground-truth for credibility labels of reviews is absent. Language Model: Workers in Amazon Mechanical Turk were tasked with writing fake reviews with the guideline of knowing all the hotel amenities in its website before writing reviews. Therefore it is quite difficult for the facet model to find contradictions or mismatch in facet descriptions. Consequently, the facet model gives marginal improvement when combined with the language model. Combined with language and consistency features, the joint model exhibits around 5% improvement in performance. Additional meta-data like the user elite and check-in status improves the performance of activity based baselines, which are not typically available for newcomers in the community. Our model using limited information (N-gram+Consistency+Activity−) performs better than the activity baselines using fine-grained information about items (like brand description) and user history. Incorporating additional user features (Activity+) further boosts its performance. Consistency Features: We remove the consistency model from the aggregated model, and see significant performance degradation of 3−4% for the Yelp∗ dataset. In the TripAdvisor dataset the performance reduction is less compared to Yelp due to reasons outlined before.