Sequence-to-Sequence Learning for Task-oriented Dialogue with Dialogue State Representation
Table 2: Automatic evaluation on test data. Best results are shown in bold. Generally, our framework outperforms other models in most automatic evaluation metrics.
['[BOLD] Model', '[BOLD] Navigation  [BOLD] BLEU', '[BOLD] Navigation  [BOLD] Macro F1', '[BOLD] Navigation  [BOLD] Micro F1', '[BOLD] Weather  [BOLD] BLEU', '[BOLD] Weather  [BOLD] Macro F1', '[BOLD] Weather  [BOLD] Micro F1']
The results show that our model outperforms other models in most of automatic evaluation metrics. In the navigation domain, compared to KV Net, we achieve 5.0 improvement on BLEU score, 37.1 improvement on Macro F1 and 27.4 improvement on Micro F1. Compared to Copy Net, we achieve 5.0 improvement on BLEU score, 41.2 improvement Macro F1 and 27.4 improvement on Micro F1. The results in navigation show our model’s capability to generate more natural and accurate response than the Seq2Seq baseline models. In the weather domain, our model generates more accurate responses than our baseline models as well. The BLEU score is a little bit lower than Copy Net and Seq2Seq with attention. This is because the forms of responses are relatively limited in weather domain. Besides, the entities in inputs are highly probable to be mentioned in responses, such as “location”. These two reasons indicate that the simpler models can capture this pattern more smoothly. The results that Seq2Seq with Attention performs better than Copy Net and KV Net also confirm this.