Unsupervised Learning of KB Queries in Task Oriented Dialogs
Table 6: Performance of various end-to-end uTOD and aTOD systems on 10 runs.
['[EMPTY]', '[BOLD] CamRest  [BOLD] Ent. F1', '[BOLD] CamRest  [BOLD] Ent. F1', '[BOLD] CamRest  [BOLD] BLEU', '[BOLD] DSTC2  [BOLD] Ent. F1', '[BOLD] DSTC2  [BOLD] Ent. F1', '[BOLD] DSTC2  [BOLD] BLEU']
The uTOD systems are trained in two different settings: (1) decoder selector trained with the gold tq and The latter setting uses heuristic to label tq and hence is trained on unannotated dialogs. It can be seen that the performance of uTODCARRL system trained using unannotated data is better than uTODMAPO, and also not far behind the aTOD system trained using annotated dialogs. In published works, aTOD systems are evaluated using results of gold KB query in the dialog context for all at turns t >tq i.e., even if the aTOD system fails to generate the correct KB query at turn t=tq, the results of the correct query will be used to predict the subsequent agent responses. As our problem is to learn a TOD system with unannotated data, we use the results of the generated KB query, even if it is incorrect, to predict the subsequent agent responses.