A Spoken Dialogue System for Spatial Question Answering in a Physical Blocks World
Table 1: Evaluation data.
['Total number of questions', '388']
Since we are continuing to refine the system, our experimental assessment remains of modest scope. We enlisted the help of 5 volunteers in our department to test the capability of the system. Among the participants were both graduate and undergraduate students, both native and non-native English speakers. The participants were instructed to ask spatial questions of the general type supported by the system, but without restriction on wording; before their first session they were shown a short demonstration of the expected kind of interaction with the system, including a few question-answer exchanges. During each evaluation session they were requested to ask between 40 and 50 questions and mark the system’s answer as correct, partially correct or incorrect. Additionally, if no answer could be given because of speech recognition errors, they were asked to indicate that as well. Finally, they were asked to indicate if the answer of the system (regardless of correctness) seemed to be improperly or oddly phrased. Each session started with the blocks positioned in a row at the front of the table. The participants were instructed to move the blocks arbitrarily to test the robustness and consistency of the spatial models.