A Dataset for Document Grounded Conversations
Table 6: The results of data analysis. LT refers to the average length of xi in scenario 1 and xi,…,xi+k in scenario 2.
['scenario', 'NW', 'LT']
We perform two kinds of automated evaluation to investigate the usefulness of the document in the conversation. The first one is to investigate if the workers use the information from the document in the conversation. The second analysis is to show that the document adds value to the conversation. Let the set of tokens in the current utterance xi be N, the set of tokens in the current section si be M, the set of tokens in the previous three utterances be H, and the set of stop words be S. In scenario 1, we calculate the set operation (NW) as |((N∩M)∖H)∖S|. Let the tokens that appear in all the utterances (xi,…,xi+k) corresponding to the current section si be K and the tokens that appear in all the utterances ( xi,…,xi+p) corresponding to the previous section si−1 be P. In scenario 2, we calculate the set operation (NW) as |((K∩M)∖P)∖S|. It also shows that they use the information to construct the responses.