Learning a Matching Model with Co-teaching for Multi-turn Response Selection in Retrieval-based Dialogue Systems
Table 1: Evaluation results on the two data sets. Numbers marked with ∗ mean that the improvement is statistically significant compared with the best baseline (t-test with p-value <0.05). Numbers in bold indicate the best strategies for the corresponding models on specific metrics.
['[EMPTY]', '[BOLD] Douban MAP', '[BOLD] Douban MRR', '[BOLD] Douban P@1', '[BOLD] Douban R10@1', '[BOLD] Douban R10@2', '[BOLD] Douban R10@5', '[BOLD] ECD MAP', '[BOLD] ECD MRR', '[BOLD] ECD P@1', '[BOLD] ECD R10@1', '[BOLD] ECD R10@2', '[BOLD] ECD R10@5']
We can see that all teaching strategies can improve the original models on both data sets, and improvement from the best strategy is statistically significant (t-test with p-value <0.05) on most metrics. On Douban, the best strategy for SMN is teaching with dynamic margins, and it is comparable with teaching with dynamic instance weighting for DAM, while on ECD, for both SMN and DAM, the best strategy is teaching with dynamic data curriculum. The difference may stem from the nature of training sets of the two data. The training set of Douban is built from random sampling, while the training set of ECD is constructed through response retrieval that may contain more false negatives. Thus, in training, Douban could be cleaner than ECD, making “hard data filtering” more effective than “soft labeling” on ECD. This somehow indicates the gap between training and test in real applications for the existing research on response selection, and thus demonstrates the merits of this work.