A Byte-sized Approach to Named Entity Recognition
Table 5: F1 scores across Bio-ID categories for byte NN model. Columns are feature(s) used. Best entity results are bolded.
['[BOLD] Entity type', '[BOLD] Bytes', '[BOLD] BPE', '[BOLD] Pre-trained BPE', '[BOLD] Pre-trained word', '[BOLD] Bytes + Pre-trained BPE']
Previous works have shown that pre-trained word embeddings are important features for word-level NER models; we find that they are less useful for byte-level models. For a consistent feature set across bytes, contiguous bytes belonging to the same word have the same word feature. This repetition of information may diminish the effectiveness of word embeddings in the byte-level models. Because the Bio-ID dataset is dominated by protein or genes, the byte NN model trained on byte and pre-trained BPE embeddings has a higher overall micro-F1 score than the byte NN model that only uses pre-trained BPE embeddings. With these results, we emphasize that BPE features are useful subword information for NER at the byte-level.