Hierarchical Multi-Label Dialog Act Recognition on Spanish Data
Table 7: Results on Level 2 using the two segment representation approaches.
['[BOLD] Approach', '[BOLD] MR  [ITALIC] m', '[BOLD] MR  [ITALIC] s', '[BOLD] Acc  [ITALIC] m', '[BOLD] Acc  [ITALIC] s', '[BOLD] P  [ITALIC] m', '[BOLD] P  [ITALIC] s', '[BOLD] R  [ITALIC] m', '[BOLD] R  [ITALIC] s', '[BOLD] F1  [ITALIC] m', '[BOLD] F1  [ITALIC] s', '[BOLD] HL  [ITALIC] m', '[BOLD] HL  [ITALIC] s']
The only exception is the Hamming loss, which, on average, is equal for both approaches. Furthermore, since we are considering less segments, the training times per epoch are reduced to 0.40 and 11.67 seconds, respectively. However, in this case, the importance of the first preceding segment is more pronounced, reducing the loss to less than a third and improving the remaining metrics by around 20 percentage points (p≈5.0e−324. This makes sense considering that the dialogs feature many question-answer pairs focused on the same target information, which is the focus of Level 2 labels. Thus, in those cases, the labels of both segments are the same. Consequently, the labels of the first preceding segment provide an important cue for the identification of those of the current segment. Using information from the current segment only leads to a slight but significant improvement (p≈0.01). However, also considering the Level 1 classification of the first preceding segment leads to an improvement around 1.5 percentage points on every non-loss metric (p≈8.7e−6). This is still explained by the presence of multiple question-answer pairs in the dialogs, as if the preceding segment is labeled as Question on Level 1, then the current segment probably has the same Level 2 labels as the preceding segment. The improvement achieved using information from additional preceding segments is not statistically significant (p≈0.74). Similarly to what happened on Level 1, the performance on user segments is different from that on system segments.