Big Bird: Transformers for Longer Sequences
Table 1: Building block comparison @512
['Model', 'MLM', 'SQuAD', 'MNLI']
As an initial sanity check, we performed basic experiments to test whether these intuitions are sufficient in getting performance close to BERT like models, while keeping attention linear in the number of tokens.