Taking Notes on the Fly Helps Language Pre-Training
Table 1: Hyper-parameters for the pre-training and fine-tuning on both BERT and BERT-TNF.
['[EMPTY]', 'Pre-training', 'Fine-tuning']
BERT-base consists of 12 Transformer layers. For each layer, the hidden size is set to 768 and the number of attention head (H) is set to 12. We use the same pre-training hyper-parameters for both BERT and BERT-TNF. All models are pre-trained for 1000k steps with batch size 256 and maximum sequence length 512. and (β1,β2) to (0.9, 0.98). The peak learning rate is set to 1e-4 with a 10k-step warm-up stage. After the warm-up stage, the learning rate decays linearly to zero. We set the dropout probability to 0.1 and weight decay to 0.01. There are three additional hyper-parameters for BERT-TNF, half window size k, discount factor λ and weight γ. We set k as 16, λ as 0.5, γ as 0.1 for the main experiment. All hyper-parameter configurations are reported in Table