Assessing Discourse Relations in Language Generationfrom Pre-trained Language Models
Table 4: % of connectives in generated texts that are consistent with human annotation, stratified by the # of annotators agreeing on the relation.
['[EMPTY]', 'Fine-tuned', 'Organic']
For the connectives since and while which can signal two relations, we count the model as correct if either relation is annotated by humans. The results reveal that wrong connective could be a prominent source of error in GPT-2 generation, though the fine-tuned model agrees better with humans. Notably, for relations that humans agree more consistently, the models also generate correct relations more often. This hints that GPT-2 captures obvious, unambiguous relations better.