Improving Response Selection in Multi-turn Dialogue Systemsby Incorporating Domain Knowledge
Table 2: Evaluation results of our models compared to various baselines on Ubuntu Dialogue Corpus.
['[BOLD] Model', '[BOLD] R2@1', '[BOLD] R10@1', '[BOLD] R10@3', '[BOLD] R10@5']
Our model outperforms all other models used as baselines. The largest improvement of our model compared to the best of the baselines (i.e. ESIM in general and SMNdyn for R2@1 metric) are with respect to the R10@1 and R10@3 metric, where we observed absolute improvements of 0.013 and 0.014 corresponding to 1.8% and 1.6% relative improvement , respectively. For R2@1 and R10@5 we observed more modest improvements of 0.007 (0.8%) and 0.005 (0.5%), respectively. Our results are significantly better with p<10−6 for a one-sample one-tailed t-test compared to the best baseline (ESIM), on R10@1, R10@3, R10@5 metrics, using the outcome of 15 independent experiments. The variance between different trials is smaller than 0.001 for all evaluation metrics. When domain knowledge is incorporated by simple addition (as in the work of Bahdanau et al. AK+-DE-biGRU, we noticed 0.5% further improvement. Note however, that the results are not as good as when using the proposed weighted addition. Finally, using our method of incorporating domain knowledge in combination with embeddings trained from scratch with fastText Bojanowski et al. , the performance gets 0.3% better than when using pretrained word2vec embeddings. In total, compared to the DE-biGRU baseline, our model (AK-DE-biGRU) achieves 10% of improvement in terms of the R10@1 metric. Thus, the results clearly suggest that both the attention mechanism and the incorporation of domain knowledge, are effective approaches for improving the dual encoder architecture. Curiously, we noticed that for the baseline models, using a GRU as the encoder is better than using a biGRU. This finding is in line with the results from Kadlec et al. However, the table is turned when augmenting the models with an attention mechanism where the biGRU-based model outperforms the one with the GRU. This observation motivates us to consider a biGRU instead of a GRU in our final model.