Learning a Matching Model with Co-teaching for Multi-turn Response Selection in Retrieval-based Dialogue Systems
Table 2: Evaluation results of co-teaching initialized with different networks.
['[EMPTY]', '[BOLD] Douban (Margin) MAP', '[BOLD] Douban (Margin) MRR', '[BOLD] Douban (Margin) P@1', '[BOLD] Douban (Margin) R10@1', '[BOLD] Douban (Margin) R10@2', '[BOLD] Douban (Margin) R10@5', '[BOLD] ECD (Curriculum) MAP', '[BOLD] ECD (Curriculum) MRR', '[BOLD] ECD (Curriculum) P@1', '[BOLD] ECD (Curriculum) R10@1', '[BOLD] ECD (Curriculum) R10@2', '[BOLD] ECD (Curriculum) R10@5']
S4SS5SSS0Px2 Answer to Q2: as a case study of co-teaching with two networks in different capabilities, we initialize model A and model B with DAM and SMN respectively, and select teaching with dynamic margins for Douban and teaching with dynamic data curriculum for ECD (i.e., the best strategies for the two data sets when co-teaching is initialized with one network). We find that co-teaching is still effective when starting from two networks, as both SMN and DAM get improved on the two data sets. Despite the improvement, it is still better to learn the two networks one by one, as co-teaching with two networks cannot bring more improvement than co-teaching with one network, and the performance of the stronger one between the two networks could also drop (e.g., DAM on Douban). We guess this is because the stronger model cannot be well taught by the weaker model, especially in teaching via “soft labels”, and as a result, it is not able to transfer more knowledge to the weaker one as well.