A Hybrid Retrieval-Generation Neural Conversation Model
Table 10. The response generation performance when we vary different distant supervision signals. This table shows the results for the setting “k’=3”, where there are 3 positive response candidates for each conversation context. “SentBLEU” denotes using sentence-level BLEU scores as distant supervision signals.
['Model Supervision', 'HybridNCM-RS BLEU', 'HybridNCM-RS ROUGE-L', 'HybridNCM-RSF BLEU', 'HybridNCM-RSF ROUGE-L']
We find that distant supervision signals like BLEU-1 are quite effective for training the hybrid ranking module. The sentence-level BLEU is not a good choice for the distant supervision signal. The reason is that the sentence-level BLEU is computed only based on the n-gram precision statistics for a given sentence pair. This score has a larger variance compared with the corpus-level BLEU. Since sentence-level BLEU scores would become very small smoothed values if there are no 4-gram or trigram matches between two sentences, which may happen frequently in short text pairs.