LRG at SemEval-2020 Task 7: Assessing the Ability of BERT and Derivative Models to Perform Short-Edits based Humor Grading
Table 2: Data distribution across the Training-Development-Test splits (number of samples)
['[BOLD] Dataset', '[BOLD] Sub-Task', '[BOLD] Training', '[BOLD] Development', '[BOLD] Test']
We validate our models with the mean-squared-error loss for the development set of sub-task 1. The official evaluation metric for sub-task 1 (humor grade regression) is the root-mean-squared-error (RMSE) loss. Whereas, the official evaluation metric for sub-task 2 (humor grade based multi-class classification) is the categorical accuracy. Since we directly use the sub-task 1 models for a zero-shot inference over the sub-task 2 data samples, there is no training involved for sub-task 2. Hence, we treat the entire sub-task 2 dataset (i.e., training + development + test) as the test set for our experiments. We also report the accuracy results on the official test set for the Humicroedit dataset. Humicroedit and FunLines datasets share a similar format, but their backgrounds differ with respect to the annotations. To check the generalization of BERT models in terms of short-edits based humor detection, we perform cross-dataset inferences. For this, we use models trained on the Humicroedit dataset for an inference over the FunLines dataset and those trained on the FunLines dataset for an inference over the Humicroedit dataset. Further, we use bert-viz tool by \newcitevig2019transformervis to get a visualization over the self-attention weights of the BERT models.