Transferability of Natural Language Inference to Biomedical Question Answering
Table 5: Experiments of the order importance in sequential transfer learning. Factoid evaluation metrics are strict accuracy (SAcc), lenient accuracy (LAcc) and mean reciprocal rank (MRR). List evaluation metrics are precision (Prec), recall (Recall) and score of macro f1 (F1). Bold denotes the best score of the columns in each task.
['Order Importance # of Task', 'Order Importance Sequence of Transfer Learning', 'Order Importance Factoid (%)', 'Order Importance Factoid (%)', 'Order Importance Factoid (%)', 'Order Importance List (%)', 'Order Importance List (%)', 'Order Importance List (%)']
To bridge the gap between different tasks, the order of sequential transfer learning is important. Thus, we build our base learning sequence as BioBERT-MNLI-SQuAD-BioASQ rather than switching the order of intermediate tasks such as BioBERT-SQuAD-MNLI-BioASQ. In order to resolve the discrepancy of context length, we give a little variation to the original experimental setting. Therefore, we extract a sentence containing the ground truth answer span and set as a total paragraph to construct a minimal context. As a result, we reduce the difference by unifying the distribution of context length in our sequential transfer learning. Due to the converted distribution of context length, we achieve speed improvements on training and inference in factoid and list type questions while achieving comparable results. For the factoid type of questions, we investigate that leveraging the MNLI dataset shows consistent improvement. On the other hand, in list type questions, the performance improves when the objective function of fine-tuned tasks are related to the BioASQ objective function. In other words, fine-tuning on the SQuAD dataset needs to be trained after the MNLI dataset.