CAsT 2019: The Conversational Assistance Track Overview
Table 5. Manual response retrieval results. These runs used the manually resolved queries.
['[BOLD] Run', '[BOLD] Group', '[BOLD] MAP', '[BOLD] MRR', '[BOLD] NDCG@3']
Manual run results. The median manual run has an NDCG@3 value of 0.361. We observe that the manual_indri query-likelihood run provided by the organizers is the median run. This indicates that models trained on the limited training data may not have generalized well. The best performing run is humanbert with an NDCG@3 value of 0.589. The humanbert run uses BERT-large as a reranking method on top of Anserini results. Similar to the automatic runs, the best performing runs all leverage BERT as a feature in reranking. We also observe the gap between the best manual and automatic runs is large, a 26% relative difference in median and 35% relative difference in the best runs.