To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks
Table 2: Test Accuracy of Roberta-base, BiLSTM, and BiLSTM with Roberta Pretrained Token Embedding when trained on the full dataset. The Δ column shows the difference between each model’s accuracy and that of Roberta-Large. For LSTM models, LSTM-n-k denotes an LSTM model with n layers and k cells. + Large or + Base indicate the use of Roberta Large or Roberta Base token embeddings, respectively. The number of parameters does not count the size of embedding table.
['Models', 'Yelp Accuracy', 'Yelp Δ', 'Sports Accuracy', 'Sports Δ', 'Electronics Accuracy', 'Electronics Δ', 'Params']
(ii) Using the pretrained RoBERTa token embeddings can further reduce the accuracy gap especially when training data is limited. For example, in the Yelp review data, a 4-layers LSTM with pretrained embeddings provides additional 3 percent gain compared to its counterparts. This suggests that the embeddings learned during pretraining RoBERTa may constitute an efficient approach for transfer learning the knowledge learned in these large MLM. We further report the accuracy metric of each model using all the training data. We observe that the accuracy gap is less than 1% on the Amazon datasets. even compared to 24 layers RoBERTa-large model. As for the Yelp dataset, the accuracy gap is within 2 percent from the RoBERTa-large model, despite an order of magnitude difference in the number of parameters.