Cloze-driven Pretraining of Self-attention Networks
Table 6: Effect of different domains and amount of data for pretraining on the on the development sets of GLUE (cf. Table 2). Results are based on the CNN base model (Table 1).
['[EMPTY]', 'train data (M tok)', 'CoLA (mcc)', 'SST-2 (acc)', 'MRPC (F1)', 'STS-B (scc)', 'QQP (F1)', 'MNLI-m (acc)', 'QNLI (acc)', 'RTE (acc)', 'Avg']
For pretraining on Common Crawl, CoLA and RTE benefit most from additional training data. The same table also shows results for News Crawl which contains newswire data. This data generally performs less well than Common Crawl, even on MRPC which is newswire. A likely reason is that News Crawl examples are individual sentences of 23 words on average which compares to several sentences or 50 words on average for Common Crawl. Mutli-sentence training examples are more effective for end-tasks based on sentence pairs, e.g., there is a 14 point accuracy gap on RTE between News Crawl and Common Crawl with 4.5B tokens. More News Crawl data is most beneficial for CoLA and STS-B.