A Sequential Matching Framework for Multi-turn Response Selection in Retrieval-based Chatbots
Table 6: Evaluation results of model ablation.
['[EMPTY]', '[BOLD] Ubuntu Corpus', '[BOLD] Ubuntu Corpus', '[BOLD] Ubuntu Corpus', '[BOLD] Ubuntu Corpus', '[BOLD] Douban Corpus', '[BOLD] Douban Corpus', '[BOLD] Douban Corpus', '[BOLD] Douban Corpus', '[BOLD] Douban Corpus', '[BOLD] Douban Corpus']
We first investigated how different parts of SCN and SAN affect their performance by ablating SCNlast and SANlast. First, we replaced the utterance-response matching module in SCN and SAN with a neural tensor Socher et al. (denoted as ReplaceM) which matches an utterance and a response by feeding their representations to a neural tensor network (NTN). The result is that the performance of the two models dropped dramatically. This is because in NTN, there is no interaction between the utterance and the response before their matching; and it is doubtful if NTN can recognize important parts in the pair and encode the information into matching. As a result, the model loses important information in the pair. Therefore, we can conclude that a good utterance-response matching mechanism is crucial to the success of SMF. At least, one has to let an utterance and a response interact with each other and explicitly highlight important parts in their matching vector. Second, we replaced the GRU in the matching accumulation modules of SCN and SAN with a multi-layer perceptron (MLP) (denoted as SCN ReplaceA and SAN ReplaceA respectively). The change led to a slight performance drop. This indicates that utterance relationships are useful in context-response matching. Finally, we only left one level of granularity, either word level or segment level, in SCN and SAN, and denoted the models as SCN with words, SCN with segments, SAN with words, and SAN with segments respectively. The results indicate that segment level matching on utterance-response pairs contributes more to the final context-response matching, and both segments and words are useful in response selection.