MOSS: End-to-End Dialog System Framework with Modular Supervision
Table 1: Performance comparison on CamRest676 among the baselines, MOSS-all, and several variants of MOSS.
['[BOLD] Model', '[BOLD] Mat', '[BOLD] Succ.F1', '[BOLD] BLEU']
The first key takeaway is that the more supervision the model has, the better the performance is. (i) KVRN < (ii) NDM ≈ (iii) LIDM < (iv) TSCP < (v) MOSS w/o DPL ≈ (vi) MOSS w/o NLU < (ix) MOSS-all. We note that this performance ranking is the same as the ranking of how much supervision each system receives: (i) KVRN only incorporates supervision from one dialog module (i.e., natural language generation); (ii, iii, iv) NDM, LIDM, TSCP incorporate supervision from two dialog modules (i.e., dialog state tracking and natural language generation); (v, vi) MOSS without dialog policy learning (MOSS w/o DPL) and MOSS without natural language understanding (MOSS w/o NLU) incorporate supervision from three dialog modules; (ix) MOSS-all incorporates supervision from all four modules and outperform all models on all three metrics. Another takeaway is that models that have access to more detailed supervision need fewer number of dialogs to reach good performance. As for language generation quality (BLEU), MOSS-all with 60% training data performs worse. We suspect that it is partially because MOSS-all with 60% training data has seen fewer number of dialogs and thus has a weaker natural language generation module. We observe a large improvement on the BLEU score. So we compare our model against TSCP and some variants of MOSS on LaptopNetwork. We augment the belief span Bt originally introduced in \citeauthorlei-etal-2018-sequicity (2018) by concatenating user act, old Bt and system act in TSCP. This augmentation makes sure that TSCP has access to the same annotations as MOSS, otherwise TSCP could hardly generate reasonable response.