Where is the context? – A critique of recent dialogue datasets
Table 1: F1 scores of the modular memorization, LSTM and TED models, as well as of the end-to-end TED model. For the results in the first row, ”initial”, we use the inferred intents without any simplifications or, in the end-to-end case, the plain text utterances. For the results in rows 3 and 4, ”simplified”, we simplify the dialogue data, as explained in §2.1. The second column indicates the setting for max_history. Appendix A.3 contains a more detailed table that also shows accuracy scores.
['[EMPTY]', '[EMPTY]', 'Memorization Training', 'Modular LSTM Training', 'Modular LSTM Testing', 'Modular TED Training', 'Modular TED Testing', 'End-to-end TED Training', 'End-to-end TED Testing']
Throughout the remaining section we simplify the training data further, until all dialogues are consistent. Note, that this resulting simplified dataset is not in any way more realistic than the original. The fact that it significantly differs from the original should only illustrate the severity of the issue of ambiguous system responses. The history independence becomes apparent when we reduce the length of dialogue history that the three models take into account. Thus, none of the policies benefit from knowing the dialogue history beyond the last user input and its preceding system action.