Big Bird: Transformers for Longer Sequences
Table 6: Classification results. We report the F1 micro-averaged score for all datasets. Experiments on smaller IMDb and Hyperpartisan datasets are repeated 5 times and the average performance is presented along with standard deviation.
['Model', 'IMDb\xa0(Maas et\xa0al.,  2011 )', 'Yelp-5\xa0(Zhang et\xa0al.,  2015 )', 'Arxiv\xa0(He et\xa0al.,  2019 )', 'Patents\xa0(Lee and Hsiang,  2020 )', 'Hyperpartisan\xa0(Kiesel et\xa0al.,  2019 )']
Document Classification We compute the fraction of the dataset that exceeds 512 tokens, i.e. the length at which the document are often truncated. Following BERT, we used one layer with cross entropy loss on top of the first [CLS] token. We see that gains of using BigBird are more significant when we have longer documents and fewer training examples. For instance, using base sized model, BigBird improves state-of-the-art for Arxiv dataset by about 5% points. On Patents dataset, there is improvement over using simple BERT/RoBERTa, but given the large size of training data the improvement over SoTA is not significant. Note that this performance gain is not seen for much smaller IMDb dataset.