Transfer Learning for Information Extraction with Limited Data
Table 3: Comparison of methods according F1-score
['Method', 'Avg. of F1']
On average, our method outperforms others notably. This is because our model exploits the efficiency of BERT trained on a large amount of data; therefore, it can potentially capture the hidden representation of data. By stacking convolution and retraining the model with 78 training documents, our model has the ability to adapt to a new domain. As a result, the model can correctly extract information on bidding documents and can improve the performance over BERT-QA of 4.55%. The improvements come from two possible reasons: (i) we employ BERT to tackle the limited number of training data and (ii) we take advantage of CNN for fine-tuning to capture local context. As can be seen, our model works stably on all tags, and usually achieves the best F1 scores in comparison with BERT-QA and n-grams-MLP. It should be noted that the feature-based method obtains 0% of F1 for tag 10 (classification of the reserved values related to the case). The reason is that n-grams features and the regular expressions may not capture well the patterns of these values (tag 10). This shows that the n-grams+MLP+regex method can be affected by the definition of regular expressions. As the result, it could not find any values for tag 10 on the test set. By contrast, our model trained on BERT can exploit rich hidden representation from large pre-trained data and with 78 training documents of bids, the model can correctly extract tag 10.