Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset
Table 5: Model performance per domain (GA: goal accuracy). Domains marked with ‘*’ are those for which the service in the test set is not present in the training set. Domains like Hotels marked with ‘**’ has one unseen and one seen service. For other domains, the service in the test set was also seen in the training set. We see that the model generally performs better for domains containing services seen during training.
['[BOLD] Domain RentalCars*', '[BOLD] Joint GA 0.086', '[BOLD] Avg GA 0.480', '[BOLD] Domain Restaurants*', '[BOLD] Joint GA 0.228', '[BOLD] Avg GA 0.558']
The model performance also varies across various domains. We observe that one of the major factors affecting the performance across domains is still the presence of the service in the training data (seen services). In most cases, the performance can be observed to degrade for domains with more unseen services. Among the unseen services, those in the ‘RentalCars’ and ‘Buses’ domain, have a very high OOV rate for slot values leading to worse performance. A low joint goal accuracy and high average goal accuracy for these two domains indicates a possible skew between the performance of different slots. Among seen services, ‘RideSharing’ domain also exhibits poor performance, since it possesses the largest number of the possible slot values across the dataset. We also notice that for categorical slots, with similar slot values (e.g. “Psychologist” and “Psychiatrist”), there is a very weak signal for the model to distinguish between the different classes, resulting in inferior performance.