Longformer: The Long-Document Transformer
Table 13: Hyperparameters of the QA models. All models use a similar scheduler with linear warmup and decay.
['Param', 'WikiHop', 'TriviaQA', 'HotpotQA']
Hyperparameters of the best configuration are listed in Tab. All other hyperparameters are similar to RoBERTa ’s. For hyperparameter search, we only tuned LR for the RoBERTa baseline and tried rates [3e-5, 5e-5, 1e-4], then used the best, which is 3e-5, for all subsequent experiments with no further tuning. We trained the Longformer-large with the best configuration once and submitted its output to the leaderboard. We ran our experiments on 32GB V100 GPUs. Small model takes 1 day to train on 4 GPUs, while large model takes 1 day on 8 GPUs.