Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-Based Chatbots
Table 3: Evaluation results on the two data sets. Numbers in bold mean that the improvement is statistically significant compared with the best baseline.
['[EMPTY]', '[BOLD] Ubuntu Corpus', '[BOLD] Ubuntu Corpus', '[BOLD] Ubuntu Corpus', '[BOLD] Ubuntu Corpus', '[BOLD] Douban Conversation Corpus', '[BOLD] Douban Conversation Corpus', '[BOLD] Douban Conversation Corpus', '[BOLD] Douban Conversation Corpus', '[BOLD] Douban Conversation Corpus', '[BOLD] Douban Conversation Corpus']
Our models outperform baselines greatly in terms of all metrics on both data sets, with the improvements being statistically significant (t-test with p-value ≤0.01, except R10@5 on Douban Corpus). Even the state-of-the-art single-turn matching models perform much worse than our models. The results demonstrate that one cannot neglect utterance relationships and simply perform multi-turn response selection by concatenating utterances together. Our models achieve significant improvements over Multi-View, which justified our “matching first” strategy. DL2R is worse than our models, indicating that utterance reformulation with heuristic rules is not a good method for utilizing context information. Rn@ks are low on the Douban Corpus as there are multiple correct candidates for a context (e.g., if there are 3 correct responses, then the maximum R10@1 is 0.33). SMNdynamic is only slightly better than SMNstatic and SMNlast. The reason might be that the GRU can select useful signals from the matching sequence and accumulate them in the final state with its gate mechanism, thus the efficacy of an attention mechanism is not obvious for the task at hand.