Chat Detection in an Intelligent Assistant: Combining Task-oriented and Non-task-oriented Spoken Dialogue Systems
Table 8: Chat detection results across the numbers of votes that the majority label obtained.
['[BOLD] #Votes', '[BOLD] #Utter.', '[BOLD] Acc.', '[BOLD] P', '[BOLD] R', '[BOLD] F1']
As expected, we see that all metrics get higher as the number of agreement among the crowd workers becomes larger. In fact, we see as much as 98% accuracy when all seven workers agree. This implies that utterances easy for humans to classify are also easy for the classifiers.