Big Bird: Transformers for Longer Sequences
Table 9: MLM BPC
['Model', 'BPC']
We find that attention based contextual representation of DNA does improve BPC, which is further improved by using longer context in BigBird.