SemEval-2020 Task 5: Counterfactual Recognition
Table 4: Performance of the baseline and official submissions on Subtask-2.
['[BOLD] Team', '[BOLD] Ranking', '[BOLD] F1', '[BOLD] Recall', '[BOLD] Precision', '[BOLD] Exact Match']
We build a conditional random field (CRF) model for sequence labeling as the baseline model for Subtask-2. This model can assign labels to each token in the input sequence by taking advantage of all input tokens and previous predictions. Specifically, same as in many name entity recognition systems, this baseline model annotates the antecedent and consequent using the B/ I/O scheme, marking whether a word is at the Beginning, Inside or Outside either the antecedent or consequent. A common set of features for each word are extracted and used to train this model, including POS tags, features of nearby words, and whether the word has an uppercase/lowercase/title flag. We received 11 official submissions to Subtask-2, and most of the submissions outperform the provided baseline model. In subtask 2, top systems take two main approaches: sequence labelling or question answering, and nearly all of them benefit from pretraining. An exception is the 6th ranked team, Anderson_Sung, that use a multi-stack, birdirectional LSTM architecture to some success