GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling
Table 8: Comparison of CoNLL03 F1 scores when various types, layers and pooling strategies of BERT are employed. “first” indicates the first sub-word embedding, “mean” and “max” refer to mean and max pooling correspondingly.
['[BOLD] BERT  [BOLD] Type', '[BOLD] BERT  [BOLD] Layer', '[BOLD] BERT  [BOLD] Pooling', '[BOLD]  [ITALIC] F1']
WordPiece is adopted to tokenize sequence in BERT, which may cut a word into pieces, such as converting “Johanson” into “Johan ##son”. Therefore, additional efforts should be taken to maintain alignments between input tokens and their corresponding labels. Three strategies are conducted to obtain the exclusive BERT embedding of each token in a sequence. Firstly, we take the first sub-word as the whole word embedding after tokenization, which is employed in the original paper of BERT Devlin et al. Mean and max poolings are used as the latter two strategies. It’s reasonable that BERT trained on large model surpasses the smaller one in most cases due to the larger model capacity and richer contextual representation. For the pooling strategy, “mean” is considered to capture more comprehensive representations of rare words than “first” and “max”, thus better average performances. Additionally, we hypothesize that the higher layers in BERT encode more abstract and semantic features, while the lower ones prefer general and syntax information, which is more helpful for our NER and Chunking tasks.