CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning
Table 3: Experimental results of different baseline methods on the CommonGen test set. The first group of models are non-pretrained models, while the second group is large pretrained models that we have fine-tuned. The best models are bold and second best ones are underlined within each metric.
['[BOLD] Model ∖ Metrics bRNN-CopyNet\xa0Gu et al. ( 2016 )', 'ROUGE-2/L 2.90', 'ROUGE-2/L 19.25', 'BLEU-3/4 5.50', 'BLEU-3/4 2.00', 'METEOR 12.70', 'CIDEr 3.99', 'SPICE 10.60', 'Coverage 42.25']
We can see that all fine-tuned pre-trained models (the lower group) outperform non-pretrained models (the upper group) with a significant margin. This is not surprising because the their pretraining objectives, including masked language modeling, word ordering, and text infilling which predicts missing words or text spans, are relevant to our task. On the other hand, we find that the key disadvantage of non-pretrained models with CopyNet still falls in the failure of using all given concepts (i.e., low coverage), which results in worse results.