SemEval-2020 Task 4: Commonsense Validation and Explanation
Table 8: Subtask C results of all the submitted systems. Those marked with * did not submit system description paper, and those marked with + means they do not include Subtask C in their system description paper.
['[BOLD] Team BUT-FIT', '[BOLD] BLEU 22.4', '[BOLD] Human Eval 1.84', '[BOLD] Team CN-HIT-IT.NLP+', '[BOLD] BLEU 9.7', '[BOLD] Human Eval 1.74']
There are in total 17 valid submissions for Subtask C. There are generally two approaches: (1) sequence-to-sequence approach, where the source side is the non-sensical statement, and the reason is the target sequence. (2) language model generation approach, which uses large-scale pretrained auto-regressive language models such as GPT-2 [radford2019language] for reason generation, where the non-sensical sentence acts as prompt. Below we describe in detail the systems and their main features. Large-scale pretrained language models such as BART and GPT-2 dominates the submissions. The two systems with the highest human evaluations, namely ANA and KaLM, use additional resources such as Wiktionary, OMCS, and other commonsense datasets. This again shows that additional knowledge from structured databases can help with the generation of the reasons. According to a further experiment of BUT-FIT, the naive baseline of “copying source sentence as the reason” can get BLEU of 17.23, which can rank No. 4 among all the submissions. This indicates that BLEU, which focuses on the surface token overlap, has difficulty in evaluating the generated text reliably. The top-performed system achieves the human evaluation score of 2.1, showing the power of pretrained language models, but considering the full score 3.0, we still have a long way to go to generate human acceptable reasons.