Taking Notes on the Fly Helps Language Pre-Training
Table 2: Performance of different models on downstream tasks. Results show that BERT-TNF outperforms BERT on not only the average GLUE score, but on the majority of individual tasks. We also list the performance of two variants of BERT-TNF. Both of them leverage the node dictionary during fine-tuning. Specifically, BERT-TNF-F uses fixed note dictionary, and BERT-TNF-U updates the note dictionary as in pre-training. Both models outperforms the baseline BERT model while perform slightly worse than BERT-TNF.
['[EMPTY]', 'MNLI', 'QNLI', 'QQP', 'SST', 'CoLA', 'MRPC', 'RTE', 'STS', 'Avg.']
TNF improves the model’s performance. BERT models are severely under-trained Therefore, training faster usually indicates better final performance given the same amount of pre-training time. We can see from the table that BERT-TNF not only outperforms BERT on the average GLUE score with a large margin (1.3 points), but also outperforms it on the majority of sub-tasks. BERT- TNF’s performance improvement against the baseline is most prominent on sub-tasks with smaller datasets. Among all 8 sub-tasks, RTE has the smallest training set which contains 2.5k training samples in total On RTE, BERT-TNF obtains the biggest performance improvement (more than 6 points) compared with the baseline. On the other two small-data sub-tasks, CoLA and MRPC, BERT-TNF also outperforms the baseline with considerable margins (4 and 2 points). This indicates that TNF pre-training can indeed provide a better initialization point for fine-tuning, especially on downstream tasks with smaller data sizes. The results show that both BERT-TNF-F and BERT-TNF-U outperform BERT. This indicates that TNF, no matter if we keep the notes at fine-tuning or not, can boost its backbone pre-training method’s performance. Moreover, we also observe that their performances are both slightly worse than BERT-TNF. We hypothesize the reason behind can be the discrepancy of the pre-training and fine-tuning data. We notice that the proportion of rare words in downstream tasks are too small (from 0.47% to 2.31%). When the data distribution of the pre-training data set is very different from the downstream data sets, notes of rare words in pre-training might be not very effective in fine-tuning.