ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation
Table 3: The attention analysis results (%).
['JDC Dataset model', 'JDC Dataset P@1', 'JDC Dataset R@1', 'JDC Dataset F1@1', 'JDC Dataset P@3', 'JDC Dataset R@3', 'JDC Dataset F1@3', 'JDC Dataset P@5', 'JDC Dataset R@5', 'JDC Dataset F1@5', 'JDC Dataset P@10', 'JDC Dataset R@10', 'JDC Dataset F1@10']
Other models, such as WSeq, HRAN and HVMN, will output the relevance score based on the attention weight for each context. Therefore we can treat it as a ranking problem. Then we calculate the precision, recall and F1 score of the top 1,3,5,10 for WSeq model, HRAN model and our ReCoSa model. We can see that the WSeq obtains the best score for P@1, R@1 and F1@1. That’s because there are 80% cases that the post is labeled as 1, and the cosine similarity can rank the explicitly similar context sentence as top 1. Though the WSeq has the best score for F1@1, it doesn’t work well for F1@3, F1@5 and F1@10. That’s because the WSeq may lose some relevant contexts which are not explicitly similar to the post but are related with the response. Compared with the HRAN and WSeq, ReCoSa performs better in most cases. Take P@3 for example, the P@3 score of ReCoSa-head3 is 26.2, which is significantly better than that of HRAN and WSeq, i.e., 24.13 and 24.27. These results indicate that the relevant contexts detected by our ReCoSa model are highly coherent with human’s judgments. Furthermore, we calculate the averaged attention distance to the response, defined as: dis2resp=N∑i=1N−i+1N+1wi, where i is the index of the context sentence si and wi is the attention weight of the ith context. The dis2resp in human label is 0.399, indicting that the distribution of human attention is approximately uniform, containing both long and short distant dependencies. The dis2resp in ReCoSa is 0.477, which is closer to human than the distance in HRAN, e.g. 0.291. That is to say, our ReCoSa model can well capture the long distant dependency as compared with traditional attention on HRED, validating the correctness of our ideas.