Natural Language Processing (almost) from Scratch
Table 11: Comparison in generalization performance for POS, CHUNK and NER tasks of the networks obtained using by combining ten training runs with different initialization.
['[BOLD] Approach', '[EMPTY]', '[BOLD] POS', '[BOLD] CHUNK', '[BOLD] NER']
Because neural networks are nonconvex, training runs with different initial parameters usually give different solutions. Voting the ten network outputs on a per tag basis (“voting ensemble”) leads to a small improvement over the average network performance. The parameters of the combining layers were then trained on the existing training set, while keeping the ten networks fixed (“joined ensemble”). This approach did not improve on simple voting. We can also observe that the performance variability among the ten networks is not very large.