Unsupervised Learning of KB Queries in Task Oriented Dialogs
Table 4: Performance of various uTOD systems on 10 runs. An oracle decoder selector was used by all systems.
['[EMPTY]', '[BOLD] CamRest  [BOLD] Ent. F1', '[BOLD] CamRest  [BOLD] Ent. F1', '[BOLD] CamRest  [BOLD] BLEU', '[BOLD] DSTC2  [BOLD] Ent. F1', '[BOLD] DSTC2  [BOLD] Ent. F1', '[BOLD] DSTC2  [BOLD] BLEU']
The aim of this experiment is to analyse the performance of the proposed query predictors on the overall system. The REINFORCE query predictor was unable to predict any valid (partial or complete) queries during test, but still achieved an F1 metric comparable to other approaches. On further inspection, we found that responses contain two types of entities: informable entities (IE) and non-informable entities (NIE) Henderson et al. IEs are those entities that the users can specify to describe their needs and NIEs are those entities that the agent retrieves from the KB to satisfy the user needs. For example, in the restaurant suggestion task – cuisine, location are IEs, while restaurant name, phone number are NIEs. Retrieving the necessary NIEs is crucial to the success of the overall system. On CamRest, REINFORCE exhibits quite low NIE F1.