IIE-NLP-NUT at SemEval-2020 Task 4: Guiding PLM with Prompt Template Reconstruction Strategy for ComVE
Table 2: Results (Accuracy) on Validation (A).
['[BOLD] Model', '[BOLD] Trial', '[BOLD] Dev', '[BOLD] Test']
Comparing with the original RoBERTa large model, the RoBERTa\scriptsize{OMCS}, equipping with the additional commonsense textual corpus, obtains an improvement of 1.6 over RoBERTa\scriptsize{Large} on the development dataset, which provides evidence that the additional textual corpus facilitates the PLM with commonsense knowledge to a certain degree. Comparing with the baselines, the models with reconstructed input based on prompt template obatin strong improvement over baselines on both the development set and test set. We observe that two types of prompt template, denoted by P1 and P2, show up almost the same performance on test set based on RoBERTa\scriptsize{OMCS}. We suppose the reason for that two prompt templates cause the same effect is the same semantic of the intention behind the different forms of the surface. And both of prompt templates offer the same hints about the task to the PLM. In the final official evaluation, we submit the prediction from RoBERTa\scriptsize{OMCS}+P2 to the leaderboard as our final result. In the group of baseline model, the RoBERTa\scriptsize{OMCS} surprisingly gets a lower score compared with RoBERTa\scriptsize{Large} on the development set. When reconstructing the original input with prompt template, it brings 0.8∼1.9 gain over baseline models, i.e. RoBERTa\scriptsize{Large} and RoBERTa\scriptsize{OMCS}, on development set. The reversal of RoBERTa\scriptsize{OMCS}’s performance on both the development and test set proves that the prompt template achieves the role of activating the potential knowledge inner PLM. Moreover, with the additional information from true statement, the models with +P+C further get improved on the development set. Though the performance of RoBERTa\scriptsize{OMCS} exhibits no advanced over RoBERTa\scriptsize{Large} on the development set, its performance shows 0.2∼0.5 gains over RoBERTa\scriptsize{Large} on test set. In the final official evaluation of Explanation Selection subtask, we commit the result of RoBERTa\scriptsize{OMCS}+P+C to the leaderboard as our final result.