[
Table 2: Precision, recall, and F1 scores of our proposed BiLM pretrained NER model (last column) and recent state-of-the-art models. We use the CNN-BiLSTM-CRF architecture for our NER model in all the experiments. Source of benchmark performance scores of datasets are: NCBI-disease: Leaman and Lu (2016); BC5CDR: Li et al. (2015); JNLPBA: GuoDong and Jian (2004); MTM-CW was proposed in Wang et al. (2018a); FFN (Feed-forward network) was proposed in Crichton et al. (2017); BiLSTM was proposed in Habibi et al. (2017). The performance scores for these NER models are referred from Wang et al. (2018a).
['[BOLD] Dataset', '[BOLD] Metric', '[BOLD] Benchmark', '[BOLD] FFN', '[BOLD] BiLSTM', '[BOLD] MTM-CW', '[BOLD] BiLM-NER']
We compare our proposed BiLM pretrained NER model with state-of-the-art NER systems such as the multi-task models of Crichton et al. From the results, we see that the approach of BiLM pretraining obtains the maximum F1 score for all the datasets. For NCBI-disease dataset, the F1 score of our model is 87.34%, which is an absolute improvement of 1.20% over the multi-task learning method of Wang et al. Similarly, for other datasets, we can see that our proposed approach outperforms other benchmark systems by a significant margin. We want to mention here that our model was trained only on the provided data for a particular dataset compared with the multi-task learning methods, which require a collection of labeled data to improve their performance. This also highlights the importance of doing pretraining of the model weights as this can improve their generalization ability on the test set.