Attending to Characters in Neural Sequence Labeling Models
Table 3: Comparison of trainable parameters in each of the neural model architectures. # total shows the total number of parameters; # noemb shows the parameter count excluding word embeddings, as only a small fraction of the embeddings are utilised at every iteration.
['[EMPTY]', 'CoNLL03 # total', 'CoNLL03 # noemb', 'FCEPUBLIC # total', 'FCEPUBLIC # noemb', 'CHEMDNER # total', 'CHEMDNER # noemb']
It is important to also compare the parameter counts of alternative neural architectures, as this shows their learning capacity and indicates their time requirements in practice. While keeping the model hyperparameters constant, the character-level models require additional parameters for the character composition and character embeddings. However, the attention-based model uses fewer parameters compared to the concatenation approach. When the two representations are concatenated, the overall word representation size is increased, which in turn increases the number of parameters required for the word-level bidirectional LSTM. Therefore, the attention-based character architecture achieves improved results even with a smaller parameter footprint.