Investigating Pretrained Language Models for Graph-to-Text Generation
Table 6: Impact of additional task-adaptive pretraining of BART-Large and T5-Large on AMR17 and AGENDA datasets. TSU and TSS refer to unsupervised and supervised pretraining on the additional task-specific data.
['[BOLD] AMR17  [BOLD] Model', '[BOLD] AMR17  [BOLD] BLEU', '[BOLD] AMR17  [BOLD] METEOR', '[BOLD] AMR17  [BOLD] chrF++']
Unsupervised pretraining helps less than supervised pretraining but also brings gains in most cases. This suggests that the performance increases after additional supervised pretraining do not only come from seeing more task-specific target texts but that the models learn how to handle graphs and the graph-to-text correspondence. Also note that we use the same number of datapoints for unsupervised and supervised experiments. It is probable that unsupervised pretraining on larger amounts of in-domain data would improve the models even more.