Blockwise Self-Attention for Long Document Understanding
Figure 6: BERT notations.
['Hyper-parameter', 'Value']
All the models follow the base setting, i.e., L=12,H=768,A=12 and are trained on the same corpus — BooksCorpus and English Wikipedia with uncased word piece tokens. We fix the number of tokens per batch B×N=131,072, i.e., if sequence length N=512 then batch size B=256, if sequence length N=1024 then batch size B=128. Moreover, the pre-training of SparseBert and BlockBert follows the RoBERTa-1seq setting , i.e., we drop the NSP (Next Sentence Prediction) task, and an input sequence is up to N tokens until it reaches a document boundary. .