Adaptive Name Entity Recognition under Highly Unbalanced Data
Table 1: Label statistic
['[BOLD] Label O', '[BOLD] Count 88791', '[BOLD] Label Geo', '[BOLD] Count 37644']
Depending on specific applications, the set of entity types might vary. However, the label set most commonly used in academic research contains four labels, including LOC(location), ORG(organization), PER(person) and MISC(miscellaneous). In this project, the given corpus was annotated with 8 types of entities: Art(artifact), Eve(event), Nat(natural phenomena), Geo(geographical entity), Gpe(geopolitical entity), Tim(time indicator), Org(organization), Per(person). In this project, we implemented Bi-LSTM-CRF architecture using Pytorch framework, and we also experimented with one traditional model using CRF (Stanford NLP) and one SOTA transformer based neural network model (BERT). The best result among above models was 0.85 F1 score on test set. However, these models all suffered from the problem of imperfect annotations (wrong annotations) and highly imbalanced dataset. reported from the whole train/test/val corpus. It could be seen that the distribution of labels is highly skewed. Specifically, the total samples of five groups (Strong class): “Geo”, “Time”, “Org”, “Per” and “Gpe” is approximately 50 times more than the total samples of the last three groups (Weak class): “Art”, “Eve” and “Nat”. This characteristic poses a considerable difficulty for almost every models as most of the features learned are biased to the Strong classes. Therefore, in latter part of this report, we will also describe an approach to solve this problem.