Semi-Supervised Sequence Modeling with Cross-View Training
Table 2: Dev set performance of multi-task CVT with and without producing all-tasks-labeled examples.
['Model', 'CCG', 'Chnk', 'NER', 'FGN', 'POS', 'Dep.']
Interestingly, multi-task learning works better in conjunction with CVT than with ELMo. We hypothesize that the ELMo models quickly fit to the data primarily using the ELMo vectors, which perhaps hinders the model from learning effective representations that transfer across tasks. During multi-task CVT, the model makes predictions about unlabeled examples across all tasks, creating (artificial) all-tasks-labeled examples, so the model does not only see one task at a time. To test the value of all-tasks-labeled examples, we trained a multi-task CVT model that only computes LCVT on one task at a time (chosen randomly for each unlabeled minibatch) instead of for all tasks in parallel.