RoBERTa: A Robustly Optimized BERT Pretraining Approach
Table 1: Comparison between static and dynamic masking for BERT\textscbase. We report F1 for SQuAD and accuracy for MNLI-m and SST-2. Reported results are medians over 5 random initializations (seeds). Reference results are from yang2019xlnet.
['[BOLD] Masking', '[BOLD] SQuAD 2.0', '[BOLD] MNLI-m', '[BOLD] SST-2']
We find that our reimplementation with static masking performs similar to the original BERT model, and dynamic masking is comparable or slightly better than static masking.