Dialogue Act Sequence Labeling using Hierarchical encoder with CRF
Table 4: Comparing accuracy of our method (Hierarchical Bi-LSTM-CRF) with other methods in the literature on SwDA dataset.
['[BOLD] Model', '[BOLD] Acc(%)']
The results show that our Hierarchical Bi-LSTM-CRF model outperforms the state-of-the-art. Our model improved the DA labeling accuracy over DRLM-Conditional model by 2.2% absolute points. In order to further analyze the results, we looked into the confusion matrix to know which labels are incorrectly/correctly assigned to utterances. Among them the most confused pairs are (sd,sv) and (aa,b) which represent (statement-non-opinion, statement-opinion) and (agree-accept, acknowledge) respectively. The total number of utterances with DA ’sd’, ’sv’, ’aa’, and ’b’ are 1317, 717, 208, and 762, respectively. 103 utterances (7.8%) with true label non-opinion were predicted incorrectly as opinion, whereas, 1155 utterances (87.7%) with true label non-opinion were predicted correctly. Similarly, 200 utterances (27.9%) with true label opinion were predicted incorrectly as non-opinion whereas 473 utterances (66%) with true label opinion were predicted correctly. On further analysis of the cause of this confusion between these two class pairs, we identified that there are utterances which were classified correctly by the model, however, they were marked incorrectly classified because of bias in the ground truth. For some of the utterances, classes were not distinguishable even by humans because of the subjectivity.