Training Neural Response Selection for Task-Oriented Dialogue Systems
Table 3: Summary of the results (R100@1 scores ×100%) with fine-tuning on all six target domains. Datasets are ordered left to right based on their size. The scores in the parentheses in the target-only, ft-direct and ft-mixed rows give the performance on the general-domain reddit test data. The scores are computed with de-duplicated inputs for semeval15 (i.e., the initial dataset links more responses to the same input), and de-duplicated answers for banking.
['[EMPTY]', 'Reddit', 'OpenSub', 'amazonQA', 'ubuntu', 'banking', 'semeval15']
Results and Discussion. First, the benefits of Reddit pretraining and fine-tuning are observed in all tasks regardless of the in-domain data size. We report large gains over the target-only model (which trains a domain-specific response selection encoder from scratch) especially for tasks with smaller training datasets (e.g., banking, semeval15). The low scores of target-only with smaller training data suggest overfitting: the encoder architecture cannot see enough training examples to learn to generalise. The gains are also present even when target-only gets to see much more in-domain input-response training data: e.g., we see slight improvements on OpenSub and amazonQA, and large gains on ubuntu when relying on the ft-direct fine-tuning variant. Using off-the-shelf sentence encoders such as use or bert directly on in-domain sentences without distinguishing the input and the response space leads to degraded performance compared even to tf-idf, or the reddit-direct baseline without in-domain fine-tuning. The importance of learning the mapping from input to response versus simply relying on similarity is also exemplified by the comparison between the map method and the simple sim method: regardless of the actual absolute performance, map leads to substantial gains over sim for all vector-based baseline models. However, even the map method cannot match the performance of our two-step training regime: we report substantial gains with our ft-direct and ft-mixed fine-tuning on top of Reddit pretraining for all target domains but one (semeval15).