Semi-supervised sequence tagging with bidirectional language models
Table 5: Comparison of CoNLL-2003 test set F1 when the LM embeddings are included at different layers in the baseline tagger.
['[BOLD] Use LM embeddings at', '[ITALIC] F1± [BOLD]  std']
We speculate that the second RNN layer in the sequence tagging model is able to capture interactions between task specific context as expressed in the first RNN layer and general context as expressed in the LM embeddings in a way that improves overall system performance. These results are consistent with Søgaard and Goldberg