Conversational Response Re-ranking Based on Event Causalityand Role Factored Tensor Event Embedding
Table 3: Comparison results before and after re-ranking
['Method NCM', 'Method history', 'Method re-ranking', 'Evaluation re-ranked (%)', 'Evaluation BLEU', 'Evaluation NIST', 'Evaluation extrema', 'Evaluation dist-1', 'Evaluation dist-2', 'Evaluation PMI', 'Evaluation length']
We compared the results by referring to the ratio of responses different from the without re-ranking method (“re-ranked”), bilingual evaluation understudy (BLEU) Papineni et al. (“extrema”) score. NIST is based on BLEU, but heavily weights less frequent N-grams to focus on content words. Vector extrema computes cosine similarity between sentence vectors of a reference and a generated response from a model. Each sentence vector es is computed by taking extrema of Skip-gram word vectors ew in each dimension d as, esd = {maxw∈sewdifewd>|minw′∈sew′d|minw∈sewdotherwise. (5) esd and ewd are the dth dimensions of es and ew respectively. Additionally, we evaluated dist Li et al. (PMI) Newman et al. Dist and PMI are used to evaluate diversity and coherency respectively. PMI between a response and a dialogue history is defined as, PMI = 1|response||response|∑wrmaxwhPMI(wr,wh). (6) wr and wh are words in the response and the dialogue history respectively. Each method used a specific NCM, a range of dialogue history used for re-ranking, and re-ranking method. Methods with “1-best” used neither re-ranking and event embedding. Those with “Re-ranking” used re-ranking but did not use event embedding. Those with “Re-ranking (emb)” used both the re-ranking and the proposed event embedding method.