Mining Implicit Relevance Feedback from User Behavior for Web Question Answering
Table 6. Comparison Results on MS Marco and WikiPassageQA datasets (all AUC metrics are percentage numbers with % omitted).
['[BOLD] Model', '[BOLD] Method', '[BOLD] MS Marco  [BOLD] 2k', '[BOLD] MS Marco  [BOLD] 5k', '[BOLD] MS Marco  [BOLD] 10k', '[BOLD] WikiPassageQA  [BOLD] 2k', '[BOLD] WikiPassageQA  [BOLD] 5k', '[BOLD] WikiPassageQA  [BOLD] 10k']
Results on the MS Marco and WikiPassageQA datasets We further apply our pre-trained model (trained on 4m FeedbackQAgbdt) from the implicit relevance feedback data to two open benchmark QA datasets, MS Marco and WikiPassageQA. We find that the queries in the MS Marco dataset are simpler than those in the DeepQA dataset, consequently, with 10k human labeled data and the BERT model , the AUC can reach as high as 94.01%. Our approach also shows improvement on this dataset, although the gain is not as large as that on the DeepQA dataset. For the BiLSTM model, the gain is around 2 points on average in terms of AUC over 2k, 5k, and 10k human labeled training data in the fine-tuning stage. For the BERT model, since the baseline is already very strong, the improvement is less than 1 point. On the WikiPassageQA dataset, our approach also shows gains, with around 1.3 points AUC gains for BERT and around 1.8 points AUC gains for BiLSTM on average over 2k, 5k and 10k labeled fine-tuning data.