Building a Conversational Agent Overnight with Dialogue Self-Play
Table 2: Comparing DSTC2 and M2M Restaurants datasets on diversity of language and dialogue flows.
['[BOLD] Metric', '[BOLD] DSTC2 (Train)', '[BOLD] M2M Rest. (Train)']
First we will investigate the claim that M2M leads to higher coverage of dialogue features in the dataset. We compare the M2M Restaurants training dialogues with the DSTC2 (Henderson et al. M2M compares favorably to DSTC2 on the ratio of unique unigrams and bigrams to total number of tokens in the dataset, which signifies a greater variety of surface forms as opposed to repeating the same words and phrases. Similarly, we count the number of unique “transitions” at the semantic frame level, defined as a pair of annotations ai,ai+1 of contiguous turns. This gives a measure of diversity of dialogue flows in the dataset. M2M has 3x the number of unique transitions per turn of the dataset. We also count unique “subdialogues”, i.e. sequences of transitions ai,ai+1,…,ai+k for k={3,5}, and observe that M2M has fewer repetitions of subdialogues compared to DSTC2.