ApplicaAI at SemEval-2020 Task 11: On RoBERTa-CRF, Span CLS and Whether Self-Training Helps Them
Table 3: Best scores on the dev set achieved with RoBERTa large model on SI task. Mean, standard deviation and maximum across 10 runs with different random seeds. Numbers in brackets indicate how many self-training iterations were used.
['[BOLD] CRF', '[BOLD] Self-train', '[BOLD] FLC-F1 (std, max)']
Models with different random seeds were trained for 60K steps with an evaluation performed every 2K steps. This is equivalent to approximately 30 epochs and per-epoch validation in a scenario without data generated during the self-training procedure. Whereas use of smaller batch size and dropout is beneficial for the initial training without noisy data, it impacts self-training phase negatively. Obviously, the largest negative impact is observed when disabling dropout during training on the small amount of manually annotated data.