Taking Notes on the Fly Helps Language Pre-Training
Table 3: Experimental results on the sensitivity of BERT-TNF’s hyper-parameter k. Results show that except the result of BERT-TNF with k=0, when fixing λ and γ, BERT-TNF with different values of k all show obvious improvements on average GLUE score compared with the baseline. As long as k is larger than 4, BERT-TNF’s performance is not very sensitive to k.
['[EMPTY]', 'Avg GLUE Score']
We also conduct experiments to check if TNF’s performance is sensitive to the introduced new hyper-parameters. Empirically, we observe the value of the half window size k are important while λ and γ are not very sensitive. We can see that using a relatively larger k is usually a better choice, which indicates that using a window with k>0, the dictionary can contain more contextual information of rare words. We also tried fixing k and tuning λ and γ. We empirically find that with λ in range (0.4,0.6) and γ in range (0.01,0.2), our method can produce similar performance. Therefore, BERT-TNF is robust to λ and γ.