Review-guided Helpful Answer Identification in E-commerce
Table 1. Overview of the datasets
['Category', '# products', '# QA', '# reviews', '[ITALIC] Lq', '[ITALIC] La']
Datasets and Evaluation Metrics We experiment with seven datasets from different product categories to validate the model effectiveness. The original question-answer pairs are from a public data collection crawled by Wan and McAuley (amazon-qa2). We also utilize the product ID in the QA dataset to align with the reviews in Amazon review dataset (amazon-review-dataset) so that the corresponding reviews of each product can be obtained. Following previous work (post-helpful; prh-net), we treat the helpful votes given by customers as a proxy of the helpfulness of each answer and model this task as a binary classification problem. Since users’ votes are not always reliable (suryanto2009quality) and people tend to upvote when they decide to vote an answer. We discard answers with less than two votes and treat the answer to be helpful if it receives helpfulness score being one (i.e. X/Y=1, Y≥2) to obtain a high standard notion of helpfulness and a more reliable dataset. However, we observe that answers with only one negative vote often provide reliable examples for unhelpful answers, they are kept in the dataset. We split the dataset in each product category into portions of 80:10:10 for training, validation, and testing respectively.