A Hybrid Retrieval-Generation Neural Conversation Model
Table 9. The response generation performance when we vary the ratios of positive samples in distant supervision.
['Model', 'Supervision # Positive', 'BLEU-1 BLEU', 'BLEU-1 ROUGE-L', 'BLEU-2 BLEU', 'BLEU-2 ROUGE-L', 'ROUGE-L BLEU', 'ROUGE-L']
We further analyze the impact of the ratios of positive/ negative training samples on the response generation performance. The value of k′ is the number of positive response candidates for each conversation context when we train the hybrid ranking module. When k′=1, we select one positive candidate from the ground truth responses in the training data, which is equivalent to the negative sampling technique. As k′ increases, we construct the positive candidates by selecting one positive sample from the ground truth responses and k′−1 positive samples from the top ranked candidates by distant supervision. We find that larger k′ can improve the response generation performance. This is reasonable since larger k′ means the model can observe more positive training samples and positive/ negative response pairs in the pairwise ranking loss minimization process. However, increasing the value of k′ also adds risks of introducing noisy positive training data. Thus, the value of k′ is a hyper-parameter, and needs to be tweaked via trial and error.