Morphosyntactic Tagging with a Meta-BiLSTM Modelover Context Sensitive Token Encodings
Table 5: Comparison of optimization methods: Separate optimization of the word, character and meta model is more accurate on average than full back-propagation using a single loss function.The results are statistically significant with two-tailed paired t-test for xpos with p<0.001 and for morphology with p<0.0001.
['[BOLD] Optimization', 'Avg. F1 Score morphology', 'Avg. F1 Score xpos']
We first compare jointly training the three model components (Meta-BiLSTM, character model, word model) to training each separately. Separate optimization leads to better accuracy for 34 out of 40 treebanks for the morphological features task and for 30 out of 39 treebanks for xpos tagging. Separate optimization out-performed joint optimization by up to 2.1 percent absolute, while joint never out-performed separate by more than 0.5% absolute. We hypothesize that separately training the models forces each sub-model (word and character) to be strong enough to make high accuracy predictions and in some sense serves as a regularizer in the same way that dropout does for individual neurons.