Semi-supervised Multitask Learning for Sequence Labeling
Table 1: Precision, Recall and F0.5 score of alternative sequence labeling architectures on error detection datasets. Dropout and LMcost modifications are added incrementally to the baseline.
['[EMPTY]', 'FCE DEV  [ITALIC] F0.5', 'FCE TEST P', 'FCE TEST R', 'FCE TEST  [ITALIC] F0.5', 'CoNLL-14 TEST1 P', 'CoNLL-14 TEST1 R', 'CoNLL-14 TEST1  [ITALIC] F0.5', 'CoNLL-14 TEST2 P', 'CoNLL-14 TEST2 R', 'CoNLL-14 TEST2  [ITALIC] F0.5']
We found that including the dropout actually decreases performance in the setting of error detection, which is likely due to the relatively small amount of error examples available in the dataset – it is better for the model to memorise them without introducing additional noise in the form of dropout. However, we did verify that dropout indeed gives an improvement in combination with the novel language modeling objective. Because the model is receiving additional information at every token, dropout is no longer obscuring the limited training data but instead helps with generalisation.