Where is the context? – A critique of recent dialogue datasets
Table 3: Training and test scores of the end-to-end TED model on 3000 training and 750 test dialogues from the Taskmaster-1 self-dialogues dataset. Results change little when max_history is reduced from 10 to 2.
['max_history', 'Training F1', 'Training accuracy', 'Testing F1', 'Testing accuracy']
To remove the knowledge base dependence, we delexicalize annotated segments Mrkšić et al. the annotated segments that occur in the utterance. We then run the same end-to-end retrieval setup as we do with MultiWOZ. Once more, we find that the resulting scores are almost history independent (see From analyzing prediction mistakes and reading the dialogues it is evident that the system responses are ambiguous in the same sense as in the MultiWOZ dataset.