Asking Clarifying Questions in Open-Domain Information-Seeking Conversations
Table 3. Performance comparison with baselines. WorstQuestion and BestQuestion respectively determine the lower and upper bounds. The superscript * denotes statistically significant differences compared to all the baselines (p<0.001).
['[BOLD] Method', '[BOLD] Qulac-T Dataset MRR', '[BOLD] Qulac-T Dataset P@1', '[BOLD] Qulac-T Dataset nDCG@1', '[BOLD] Qulac-T Dataset nDCG@5', '[BOLD] Qulac-T Dataset nDCG@20', '[BOLD] Qulac-F Dataset', '[BOLD] Qulac-F Dataset MRR', '[BOLD] Qulac-F Dataset P@1', '[BOLD] Qulac-F Dataset nDCG@1', '[BOLD] Qulac-F Dataset nDCG@5', 'nDCG@20']
Oracle question selection: performance. Here we study the performance of an oracle model, i.e. assuming that an oracle model is aware of the answers to the questions. The goal is to show to what extent clarifying questions can improve the performance of a retrieval system. This shows the high potential gain of asking good clarifying questions on the performance of a conversational system. Particularly, we examine the relative improvement of the system after asking only one question and observe that BestQuestion achieves over 100% relative improvement in terms of different evaluation metrics (MRR: 0.2820→0.5677, P@1: 0.1933→0.4986, nDCG@1: 0.1460→0.3988, nDCG@5: 0.1503→0.2793, nDCG@20: 0.1520→0.2265). It is worth mentioning that we observe the highest relative improvements in terms of nDCG@1 (=173%) and P@1 (=158%), exhibiting a high potential impact on voice-only conversational systems. Question selection. We see that all models outperform OriginalQuery, confirming that asking clarifying questions is crucial in a conversation, leading to high performance gain. For instance, compared to OriginalQuery, a model as simple as σ-QPP achieves a 31% relative improvement in terms of MRR. Also, NeuQS consistently outperforms all the baselines in terms of all evaluation metrics on both data splits. All the improvements are statistically significant. Moreover, NeuQS achieves a remarkable improvement in terms of both P@1 and nDCG@1. These two evaluation metrics are particularly important for voice-only conversational systems where the system must return only one result to the user. The obtained improvements highlight the necessity and effectiveness of asking clarifying questions in a conversational search system, where they are perceived as natural means of interactions with users.