Schema-Guided Natural Language Generation
(a) Seen services.
['[BOLD] Service', '[BOLD] % Test Refs', 'Seq2Seq  [BOLD] BLEU', 'Seq2Seq  [BOLD] SER', 'CVAE  [BOLD] BLEU', 'CVAE  [BOLD] SER', 'GPT2  [BOLD] BLEU', 'GPT2  [BOLD] SER']
Next, we are interested to see how our models perform on specific services in the dataset. Recall that the original dataset consists of a set of services that can be grouped into domains: e.g., services restaurant_1 and restaurant_2 are both under the restaurant domain. Based on this, we segment our test set into three parts, by service: seen, or services that have been seen in training, partially-unseen, or services that are unseen in training but are part of domains that have been seen, and fully-unseen where both the service and domain are unseen. We sort services by how many references we have for them in test; events_1 for example constitutes 19% of the 20K test references. To focus our discussion here, we show only the top-3 and bottom-1 services in terms of percentage of test references. For fully-unseen we show the only available service (alarm_1). We show the best scores in bold and the worst scores in italic.