Few-Shot Natural Language Generation by Rewriting Templates
Table 3: Performance of models on seen and unseen domains of the SGD-NLG dataset.
['[BOLD] Approach', '[BOLD] Seen  [BOLD] BLEU', '[BOLD] Seen  [BOLD] SER', '[BOLD] Unseen  [BOLD] BLEU', '[BOLD] Unseen  [BOLD] SER']
The ideal NLG model should be able to handle domains it was not exposed to during training. In practice, this is very hard to obtain. The SGD-NLG dataset, which features unseen domains in the evaluations sets, let’s us asses this zero-shot capability of a model. Firstly, all models exhibit low SER scores in both seen and unseen domains, with the template approach being the lowest. This suggests that pre-trained language models are adept at copying and the skill generalizes to out-of-domain examples as well. This result also hints at the need for better evaluation metrics for text generation.