LRG at SemEval-2020 Task 7: Assessing the Ability of BERT and Derivative Models to Perform Short-Edits based Humor Grading
Table 3: Post Evaluation Experiments (*final submission)
['[BOLD] BERT DERIVATIVE MODEL', '[BOLD] TRAINING DATASET', '[BOLD] MASKED-LM FINE TUNING', '[BOLD] HUMICROEDIT  [BOLD] TASK-1 (TEST SET) (RMSE)', '[BOLD] HUMICROEDIT  [BOLD] TASK-2 (TEST SET) (ACC)', '[BOLD] HUMICROEDIT  [BOLD] TASK-2 (ALL DATA) (ACC)', '[BOLD] FUNLINES  [BOLD] TASK-1 (TEST SET) (RMSE)', '[BOLD] FUNLINES  [BOLD] TASK-2 (ALL DATA) (ACC)']
For the final submission, we use BERT with masked word language model fine-tuning for both the sub-tasks. On the final test-set leaderboard, our system ranks 14th for sub-task 1, with a RMSE loss of 0.53318 and 13th for sub-task 2, with an accuracy of 0.62177. Further, in the post-evaluation experiments, we analyze the performance of all the BERT models on both the Humicroedit and the FunLines dataset. Overall, the BERT models perform better without masked word language model fine-tuning. This shows the capability of the BERT models to perform humor-grading tasks without the need for any language model fine-tuning on the humor dataset. All the BERT models perform significantly well for the cross-dataset inferences, showing their generalization capabilities across datasets from different backgrounds. Smaller BERT models (ALBERT: 11M parameters and DistilBERT: 65M parameters) do not show a significant drop in performance for sub-task 1 as compared to their bigger counterparts (BERT: 110M parameters and RoBERTa: 125M parameters). But we see a relatively lower accuracy on sub-task 2 for these smaller models. For sub-task 1, BERT slightly outperforms all the other models on both the datasets. Whereas for sub-task 2, RoBERTa gives a better performance than the rest of the models, with a significant gain on the FunLines dataset. Overall, all of the BERT models perform better on the Humicroedit dataset, which might be attributed to its relatively larger size as compared to the FunLines dataset.