Review-guided Helpful Answer Identification in E-commerce
Table 2. Comparison of the F1 and AUROC scores between RAHP-Base, RAHP-NLI and comparative models
['[EMPTY]', 'Electronics F1', 'Electronics AUC', 'Sports F1', 'Sports AUC', 'Health F1', 'Health AUC', 'Home F1', 'Home AUC', 'Patio Lawn F1', 'Patio Lawn AUC', 'Phones F1', 'Phones AUC', 'Toy & Games F1', 'Toy & Games AUC']
Overall, our proposed methods substantially and consistently outperform those baseline methods in all domains. Concretely, we can observe that answer selection models generally provide strong baselines for the concerned helpfulness prediction task. Especially models with advanced architecture (e.g. ESIM model) achieves good performance due to the fact that those complicated models explicitly consider the fine-granularity relations between QA pairs. However, RAHP-Base consistently outperforms them, which demonstrates the effectiveness and necessity of taking relevant reviews into consideration. However, the improvements vary from category to category. To gain some insight of what factor causes such difference, we investigate the ratio of the overlapping vocabularies between datasets of several product categories with the SNLI dataset, since the domain difference is often a key factor in transfer learning (conversation-google). We can see that there exists a trend between the overlapping ratio and the relative improvements: larger overlapping ration generally leads to a larger improvement. For example, the performance on the Phone category has been improved for about 3.5% with the overlapping ratio of 40%, while the performance of the Home category with the overlapping ratio being 18% almost does not change. This result also suggests that one effective approach for further improving the performance, especially for categories with low performance, can be achieved by providing additional in-domain labeled data of those categories.