A dataset for resolving referring expressions in spoken dialogue via contextual query rewrites (CQR)
Table 4: Crowd-sourced data collection effort.
['Reformulations per gold', '5']
We presented crowd-sourced workers with dialogues with highlighted basis utterance and the list of slot values that we judged relevant. Workers were encouraged to use their own strategies to rewrite the basis utterance in order to increase linguistic variation. Examples, instructions, and feedback to the workers helped ensure that semantic similarity (regarding which slot values to include) was as high as possible.