Response-Anticipated Memory for On-Demand Knowledge Integration in Response Generation
Table 4: Similarity between important document tokens picked by gold responses and the accumulated attention weights in the models.
['[EMPTY]', 'Top10 tokens Emb-M', 'Top10 tokens Emb-B', 'Top20 tokens Emb-M', 'Top20 tokens Emb-B']
In this section, we investigate whether anticipating response contributes to building a better document memory. We first calculate the semantic similarity between each document token and the response using their Glove embeddings, and select top K document tokens. Next, we accumulate the attention weights of each token in all attention distributions in the self-attention weights A in Eq. Then we select the top K tokens according to their accumulated attention weights. Here, we set K=10,20. We apply metrics in liu2016not to calculate the similarity of two token sets extracted above, including maximal tokens-tokens embedding similarity (Emb-M) and bag-of-word embedding similarity (Emb-B). A higher similarity score indicates more response information anticipated by the model. Results demonstrate that our model is able to output more response-anticipated self-attention distributions, which benefits generating a response close to the ground truth.