Neural Responding Machine for Short-Text Conversation
Table 2: The results of evaluated methods. Mean score is the average value of annotated scores over all annotations. (Rtr.-based means the retrieval-based method)
['[BOLD] Models', '[BOLD] Mean Score', '[BOLD] Suitable (+2)', '[BOLD] Neutral (+1)', '[BOLD] Unsuitable (0)', '[BOLD] Agreement']
Our test set consists of 110 posts that do not appear in the training set, with length between 6 to 22 Chinese words and 12.5 words on average. Except the SMT-based model, the value of agreement is in a range from 0.2 to 0.4 for all the other models, which should be interpreted as “Fair agreement”. The SMT-based model has a relatively higher kappa value 0.448, which is larger than 0.4 and considered as “Moderate agreement”, since the responses generated by the SMT often have the fluency and grammatical errors, making it easy to reach an agreement on such unsuitable cases. This observation confirms with our intuition that the STC dataset, with one post potentially corresponding to many responses, can not be simply taken as parallel corpus in a SMT model. Surprisingly, more than 60% of responses generated by all the three NRM are labeled as “Suitable” or “Neutral”, which means that most generated responses are fluent and semantically relevant to post. Among all the NRM variants