Unsupervised Learning of KB Queries in Task Oriented Dialogs
Table 3: Performance of CARRL and other weakly supervised approaches on CamRest and DSTC2 on 10 runs. PIQ (Partial Intent Query) ratio is the fraction of generated queries that captured only a part of the user request.
['[EMPTY]', '[BOLD] Accuracy  [BOLD] DSTC2', '[BOLD] Accuracy  [BOLD] CamRest', '[BOLD] Total Test Rewards  [BOLD] DSTC2', '[BOLD] Total Test Rewards  [BOLD] CamRest', '[BOLD] PIQ Ratio  [BOLD] DSTC2', '[BOLD] PIQ Ratio  [BOLD] CamRest']
For a fair comparison, we predict the query at the same turn at which the gold query was annotated. CARRL outperforms MAPO on both the datasets. On both daatsets, CARRL has almost 50% drop in PIQs compared to MAPO.