NNE: A Dataset for Nested Named Entity Recognition in English Newswire
Table 1: Number of spans at each layer of nesting with their most frequent categories.
['[BOLD] Depth', '[BOLD] Number', '%', '[BOLD] Three most frequent categories']
The resulting NNE dataset includes a large number of entity mentions of substantial depth, with more than half of mentions occurring inside another mentions. Of the 118,525 top-level entity mentions, 47,020 (39.6%) do not have any nested structure embedded. The remaining 71,505 mentions contain 161,270 mentions, averaging 2.25 structural mentions per each of these top-layer entity mentions. Note that one span can be assigned multiple entity types. For example, the span ‘1993’ can be annotated as both \DATE and \YEAR. In NNE, 19,144 out of 260,386 total spans are assigned multiple types. To measure how clearly the annotation guidelines delineate each category, and how reliable our annotations are, inter-annotator agreement was calculated using annotations on Section 02, which was annotated by all four annotators. An adjudicated version was created by deciding a correct existing candidate label from within the four possibilities, or by adjusting one of them on a token level. For the purposes of inter-annotator agreement, a tag stack is calculated for each word, essentially flattening each token’s nested annotation structure into one label. O_O. Agreement using Fleiss’ kappa over all tokens is 0.907. Considering only tokens that are part of at least one mention according to at least one annotator, Fleiss’ kappa is 0.832. Average precision, recall and F1 score across four annotators with respect to the adjudicated gold standard are 94.3, 91.8 and 93.0.