Natural Language Processing (almost) from Scratch
Table 13: Generalization performance of our neural network architecture trained with our language model (LM2) word embeddings, and with the word representations derived from the Brown Clusters. As before, all networks are fed with a capitalization feature. Additionally, POS is using a word suffix of size 2 feature, CHUNK is fed with POS, NER uses the CoNLL 2003 gazetteer, and SRL is fed with levels 1–5 of the Charniak parse tree, as well as a verb position feature. We report performance with both convex and non-convex architectures (300 hidden units for all tasks, with an additional 500 hidden units layer for SRL). We also provide results for Brown Clusters induced with a 130K word dictionary, as well as Brown Clusters induced with all words of the given tasks.
['[BOLD] Approach', '[BOLD] POS', '[BOLD] CHUNK', '[BOLD] NER', '[BOLD] SRL']
Our embeddings perform at least as well as Brown Clusters. Results are more mitigated in a convex setup. For most task, going non-convex is better for both word representation types. In general, “fine-tuning” our embeddings for each task also gives an extra boost.