Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
Table 7: Hyper-parameters for pretraining.
['[BOLD] Hparam', '[BOLD] Base Scale', '[BOLD] Large Scale']
One exception is the learning rate used for B10-10-10H1024 at the base scale. Specifically, we find the training can be unstable when the depth goes beyond 24 layers (in the case of B10-10-10H1024) at base scale, especially for the MLM objective. Hence, we reduce the learning to 8e-5 for the B10-10-10H1024 F-TFM during base-scale pretraining. This has a side effect of a slower training pace and potentially a slightly worse finetuning performance. However, we does not observe such instability when the batch size is increased such as in the large-scale setting.