Longformer: The Long-Document Transformer
Table 4: Top: changing window size across layers. Bottom: with/without dilation (@ 150K steps on phase1)
['Model', 'Dev BPC']
The top of Tab. We observe that increasing the window size from the bottom to the top layer leads to the best performance, arranging them in the reverse way leads to worse performance, and using a fixed window size (the average of window sizes of the other configuration) leads to a performance that it is in between. The bottom of Tab. Adding some dilation to two heads leads to some improvement compared with no dilation at all. We ran the small model experiments on 4 RTX8000 GPUs for 16 days. For the large model, we ran experiments on 8 RTX8000 GPUs for 13 days. Most of our hyperparameter search is similar to the ablation in Tab. We experimented with absolute position embeddings and learned position embeddings, dropout values of [0.1, 0.2] (small model) and [0.1, 0.4] (large model), pre-layernorm and post-layernorm Xiong et al. Number of gradient updates/phase reported in Tab.