Addressee and Response Selection in Multi-Party Conversations with Speaker Interaction RNNs
Table 2: Addressee and response selection results on the Ubuntu Multiparty Conversation Corpus. Metrics include accuracy of addressee selection (ADR), response selection (RES), and pair selection (ADR-RES). RES-CAND: the number of candidate responses. T: the context length.
['[EMPTY]', '[EMPTY]', 'RES-CAND = 2 DEV', 'RES-CAND = 2 TEST', 'RES-CAND = 2 TEST', 'RES-CAND = 2 TEST', 'RES-CAND = 10 DEV', 'RES-CAND = 10 TEST', 'RES-CAND = 10 TEST', 'RES-CAND = 10 TEST']
For fair and meaningful quantitative comparisons, we follow \citeauthorouchi-tsuboi:2016:EMNLP2016 \shortciteouchi-tsuboi:2016: EMNLP2016’s evaluation protocols. SI-RNN improves the overall accuracy on the addressee and response selection task. Two ablation experiments further analyze the contribution of role-sensitive units and joint selection respectively. We then confirm the robustness of SI-RNN with the number of speakers and distant responses. Finally, in a case study we discuss how SI-RNN handles complex conversations by either engaging in a new sub-conversation or responding to a distant message. Overall Result. In particular, addressee selection (ADR) benefits most, with different number of candidate responses (denoted as RES-CAND): around 12% in RES-CAND =2 and more than 10% in RES-CAND =10. Response selection (RES) is also improved, suggesting role-sensitive GRUs and joint selection are helpful for response selection as well. The improvement is more obvious with more candidate responses (2% in RES-CAND =2 and 4% in RES-CAND =10). These together result in significantly better accuracy on the ADR-RES metric as well. Ablation Study. First, we share the parameters of IGRUS/IGRUA/GRUO. The accuracy decreases significantly, indicating that it is crucial to learn role-sensitive units to update speaker embeddings. Second, to examine our joint selection, we fall back to selecting the addressee and response separately, as in Dynamic-RNN. We find that joint selection improves ADR and RES individually, and it is particularly helpful for pair selection ADR-RES. Number of Speakers. Numerous speakers create complex dialogs and increased candidate addressee, thus the task becomes more challenging. Recent+TF-IDF always chooses the most recent speaker and the accuracy drops dramatically as the number of speakers increases. Direct-Recent+TF-IDF shows better performance, and Dynamic-RNNis marginally better. SI-RNN is much more robust and remains above 70% accuracy across all bins. The advantage is more obvious for bins with more speakers. Addressing Distance. Addressing distance is the time difference from the responding speaker to the ground-truth addressee. It is important to note that for those distant responses, Dynamic-RNN sees a clear performance decrease, even worse than Direct-Recent+TF-IDF. In contrast, SI-RNN handles distant responses much more accurately.