Is this Dialogue Coherent? Learning from Dialogue Acts and Entities
Table 3: Average (5 runs) of Accuracy (Acc.), Mean Reciprocal Rank (MRR) and Recall at one (R@1) and two (R@2) for response selection using both data generation methodologies (Internal and External Swap) on Switchboard.
['[EMPTY]', '[BOLD] Internal Swap  [BOLD] Acc.', '[BOLD] Internal Swap  [BOLD] MRR', '[BOLD] Internal Swap  [BOLD] R@1', '[BOLD] Internal Swap  [BOLD] R@2', '[BOLD] External Swap  [BOLD] Acc.', '[BOLD] External Swap  [BOLD] MRR', '[BOLD] External Swap  [BOLD] R@1', '[BOLD] External Swap  [BOLD] R@2']
Response selection Neural models seem to capture better turn-level coherence compared to classic grid SVM-based approaches. In both data generation methodologies, Internal (IS) and External Swap (ES), SVM coherence models are outperformed by neural ones for all metrics considered. As expected, entity features (entrole, entword) play a more prominent role in ES compared to IS. In both cases, entity features seem to be better captured by neural models relying on our proposed input representation. When considering lexical information (entword), however, entrole features seem less relevant. This might be due to the fact that spoken dialogue has usually less complex syntactic structures compared to written text. Furthermore, parsers are usually trained on written text, and thus might be more error-prone when applied to dialogue where there are disfluencies, sudden changes of topics, etc. We notice that DAs alone (without entity information) play an important role in both IS and ES. Turn features capturing speaker information seem helpful for both DAs and entities. In general, the combination of DAs and entities gives the best results both in SVM and neural models for both tasks, with the best performing one being the model combining entword, DA and turn features and without entrole. Additionally, if we compare the IS setting to ES in terms of best MRR, Accuracy and Recall, the former seems more difficult. This confirms our expectations that IS might be an harder task for coherence.