CNRL at SemEval-2020 Task 5: Modelling Causal Reasoning in Language with Multi-Head Self-Attention Weights based Counterfactual Detection
Table 3: Post evaluation analysis of various transformer models in base architecture
['[BOLD] Transformer Model', '[BOLD] Sub-Task 1  [BOLD] F1', '[BOLD] Sub-Task 1  [BOLD] Recall', '[BOLD] Sub-Task 1  [BOLD] Precision', '[BOLD] Sub-Task 2  [BOLD] F1', '[BOLD] Sub-Task 2  [BOLD] Recall', '[BOLD] Sub-Task 2  [BOLD] Precision']
For the final submission, we use RoBERTa and BERT as the transformer model in our base architecture for sub-task 1 and sub-task 2 respectively. On the final task leaderboard, our system ranks 13th (0.845 F1) for the counterfactual detection sub-task and 7th (0.688 F1) for the antecedent-consequent detection sub-task. Since we treat antecedent-consequent detection as a regression task, we do not monitor the Exact Match score between the predicted and ground truth spans, which is more significant for token-level sequence labelling based approaches. RoBERTa gives the best results for the counterfactual detection sub-task. Whereas, BERT gives the best results for the antecedent-consequent detection sub-task. DistilBERT, a considerably smaller model (65M parameters) shows comparable performance with the rest of the transformer models (110M+ parameters) for both the sub-tasks.