Longformer: The Long-Document Transformer
Table 7: Average and 95th percentile of context length of datasets in wordpieces. WH: WikiHop, TQA: TriviaQA, HQA: HotpotQA, ON: OntoNotes, HY: Hyperpartisan news
['Wordpieces', 'WH', 'TQA', 'HQA', 'ON', 'IMDB', 'HY']
We apply Longformer to multiple long document tasks, including QA, coreference resolution and classification. Tab. Our primary goal is to evaluate whether our attention mechanism can act as a replacement for the standard self-attention mechanism in BERT style models, and to perform controlled trials against a strong baseline. We are also interested in evaluating whether we can replace complicated task specific models necessitated by BERT’s limited context with simpler models that just concatenate all available context into a single sequence.