Parallel sequence tagging for concept recognition
Table 4: Precision and recall for unseen concepts in the test set.
['Annotation set', 'Harmonisation', 'proper ref.\xa0count', 'proper ref.\xa0count', 'proper prec.', 'proper recall', 'extended ref.\xa0count', 'extended ref.\xa0count', 'extended prec.', 'extended recall']
The ids-only strategy is omitted in the table, as this configuration can never predict unseen concepts. The spans-only and spans-first strategies systematically yield identical results, as they only differ in cases where the latter backs off to ID predictions, which have been filtered out in this evaluation. With the ids-first strategy, many span predictions for unseen concepts are shadowed by an ID prediction for a concept known from the training set (which is then ignored in this specific evaluation). For some annotation types (e. g. CHEBI[_EXT], GO_BP[_EXT], SO[_EXT]), this removal improves precision, i. e. more false positives than true positives were removed. In other cases, precision suffers from the removal; recall decreases in all cases. The diversity of the individual annotation sets shows even more clearly when it comes to predicting unseen concepts. In general, the level of precision and recall for unseen concepts varies greatly across annotation sets, as does the number of unseen concepts in the reference (cf. There is a loose negative correlation to the performance on the entire test set: annotation sets like NCBITaxon[_EXT] and SO[_EXT] show high overall scores and low scores for unseen concepts, whereas more difficult sets like PR[_EXT] have comparatively high precision and recall for unseen concepts. A possible explanation is that the former have little variability and a high overlap between training and test set, leading to a strong bias for known concepts (overfitting tendency), which is beneficiary for the test set as a whole, but not for the subset of unseen concepts. The latter show great variability of concept labels and surface names, which makes the task harder but also leads to better generalisation.