Big Bird: Transformers for Longer Sequences
Table 17: GLUE Dev results on base sized models. Number of training examples is reported below each task. MCC score is reported for CoLA, F1 score is reported for MRPC, Spearman correlation is reported for STS-B, and accuracy scores are reported for the other tasks.
['System', 'MNLI-(m/mm)', 'QQP', 'QNLI', 'SST-2', 'CoLA', 'STS-B', 'MRPC', 'RTE']
The General Language Understanding Evaluation (GLUE) benchmark We used the same training parameters as mentioned in https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.glue.md. Our model parameters are b=64,g=2×b, w=3×b,r=3×b ( we used the BigBird-itc base model pretrained on MLM task). We compare the performance of BigBird to BERT, XLNet We find that even on task that have a much smaller context, our performance is competitive to full attention models.