Exploring and Predicting Transferability across NLP Tasks
Table 5: Fine-tuning results for question answering tasks.
['Task', '[BOLD] Data regime  [BOLD] full', '[BOLD] Data regime  [BOLD] full', '[BOLD] Data regime  [BOLD] full', '[BOLD] Data regime  [BOLD] full', '[BOLD] Data regime  [BOLD] full', '[BOLD] Data regime  [BOLD] full', '[BOLD] Data regime  [BOLD] limited', '[BOLD] Data regime  [BOLD] limited', '[BOLD] Data regime  [BOLD] limited']
For both fine-tuning and transfer learning, we use the same architecture across tasks, apart from the task-specific output layer. The feature extractor, i.e., BERT, is pretrained while the task-specific output layer is randomly initialized for each task. All the parameters are fine-tuned end-to-end. An alternative approach is to keep the feature extractor frozen during fine-tuning. We find that fine-tuning the whole model for a given task leads to better performance in most cases, except for WNLI and DROP, possibly because of their adversarial nature (see In our experiments, we follow the fine-tuning recipe of JDevlin19, i.e., only fine-tuning for a fixed number of t epochs for each class of problems. We develop our infrastructure using the HuggingFace’s Transformers TWolf19 and use the hyperparameters recommended by the library for each class. We perform experiments in two data regimes: full and limited. In the full regime, all training data for the associated task is used while in the limited setting, we artificially limit the amount of training data by randomly selecting 1K training examples without replacement following JPhang18. For each experiment in the limited regime, we perform 20 random restarts (1K examples are resampled for each restart) and report the mean and standard deviation. We show the results after each training epoch t.