Assessing Discourse Relations in Language Generationfrom Pre-trained Language Models
Table 5: Consistency between human annotated and predicted discourse relations, measured in macro-f1 of the four relation types. (≥n): ≥n annotators agree on a relation. (*): p<0.05 on a bootstrapping test. Accuracies tabulated in Appendix C.
['[EMPTY]', 'Fine-tuned GPT-2', 'Fine-tuned predicted', 'Organic GPT-2', 'Organic predicted']
For both fine-tuned and organic cases, the predicted connective aligns closer to human labels than those generated by GPT-2. The improvement is notably more substantial for the organic case, an indication that fine-tuning GPT-2 nudges the model very close to what the connective prediction model learns. The overall improvement is likely due the connective prediction model having access to text before and after the connective, while the initial language generation model does not. This finding points to future work on considering stronger discourse-related signals Ji et al.