Towards Better Understanding of Spontaneous Conversations: Overcoming Automatic Speech Recognition Errors With Intent Recognition
Table 2: Rescoring accuracy w.r.t. intent length measured on an annotated dataset of 500 rescored intents.
['[BOLD] Intent length', '[BOLD] Occurrences', '[BOLD] Accuracy [%]']
To evaluate the effectiveness of the proposed algorithm, we have sampled a dataset of 500 rescored intent annotations found in the lattices in cancellations and refunds domain. The correctness of the rescoring was judged by two annotators, who labeled 250 examples each. The annotators read the whole conversation transcript and listened to the recording to establish whether the rescoring is meaningful. In cases when a rescored word was technically incorrect (e.g., mistaken tense of a verb), but the rescoring led to the recognition of the correct intent, we labeled the intent annotation as correct. Please note that every result above 50% indicates an improvement over the ASR best path recognition, since we correct more ASR errors than we introduce new mistakes.