Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation
Table 3: Fine-grained analysis of character-level entropy. In the seq2BF+ model, we analyze the average entropy of keywords and remaining words separately.
['[BOLD] Model', 'seq2seq', 'seq2BF−', 'seq2BF+ keyword', 'seq2BF+ remaining']
We find that, provided with a noun term, seq2BF can generate meaningful remaining words (keyword excluded) with an entropy of 7.422, higher than 6.971 given by seq2BF without keywords. Noticing that the seq2BF model is exactly the same in content-introducing and non-content-introducing settings, we believe the second conjecture holds. Choosing the most likely reply yields universal utterances; moreover, RNN sequence generators are reluctant to introduce concrete concepts, provided with one or a few universal words/terms (like I and you) that are greedily chosen at the beginning.