SemEval-2020 Task 5: Counterfactual Recognition
Table 3: Performance of the baseline and official submissions on Subtask-1.
['[BOLD] Team', '[BOLD] Ranking', '[BOLD] F1', '[BOLD] Recall', '[BOLD] Precision']
The baseline used for Subtask-1 is a simple SVM classifier with the linear kernel function. In the baseline model, we first take some basic preprocessing steps starting with lemmatization; we then extract term frequency and inverse document frequency (tf-idf) features for training the SVM model to perform binary classification. The motivation behind using this simple SVM as a baseline is to create a simple model that can identify counterfactuals by learning and searching for the presence keywords and phrases like “had” or “should have been”, that tend to mark the presence of a counterfactual in a number of counterfactual grammatical forms. The baseline has poor performance, signalling that most counterfactuals cannot be determined based on the presence of certain words and that reasoning is necessary. We received 27 submissions to Subtask-1. Their system struggles with data imbalance and they conclude that transformer networks can improve their performance. Among the top 8 competitors, BERT and RoBERTa based systems are most popular, being used as the primary models or as a part of their final ensemble in 5 and 4 of the top 8 participants’ systems respectively. XLNet and ALBERT are less popular choices, but they are also used in the first and second of the top 8 participating systems respectively.