Partially-Typed NER Datasets Integration: Connecting Practice to Theory
Table 2: Performance Comparison. Standard♠ (S sent. w. N types) is a conventional sequence labeling model trained with s sentences (fully annotated with n types of entities).
['Dataset', 'Method', 'Per-type  [ITALIC] F1 cell-type', 'Per-type  [ITALIC] F1 DNA', 'Per-type  [ITALIC] F1 protein', 'Per-type  [ITALIC] F1 cell-line', 'Per-type  [ITALIC] F1 RNA', 'Micro  [ITALIC] F1']
Also, the performance of partially-typed datasets keeps dropping as the number of types grows (since larger N results in less annotated entities per type). Among these methods, Concat performs much worse than all other methods due to the effect of false negatives, which verifies the importance of properly integrating datasets. Propagate adds more constraints to the target label set of Partial and achieves worse performance (Partial achieves 2.34%, 4.07%, 6.77% gain over Propagate for n=3,4,5, respectively). Such result indicate that it is more profitable not to constraints on the target label space and allow the algorithm to infer the labels in an adaptive manner.