Semi-Supervised Sequence Modeling with Cross-View Training
Table 1: Results on the test sets. We report the mean score over 5 runs. Standard deviations in score are around 0.1 for NER, FGN, and translation, 0.02 for POS, and 0.05 for the other tasks. See the appendix for results with them included. The +Large model has four times as many hidden units as the others, making it similar in size to the models when ELMo is included. * denotes semi-supervised and † denotes multi-task.
['2*Method', 'CCG', 'Chunk', 'NER', 'FGN', 'POS', 'Dep. Parse', 'Dep. Parse', 'Translate']
only TagLM and ELMo are semi-supervised. These methods first train an enormous language model on unlabeled data and incorporate the representations produced by the language model into a supervised classifier. Our base models use 1024 hidden units in their LSTMs (compared to 4096 in ELMo), require fewer training steps (around one pass over the billion-word benchmark rather than many passes), and do not require a pipelined training procedure. Therefore, although they perform on par with ELMo, they are faster and simpler to train. Increasing the size of our CVT+Multi-task model so it has 4096 units in its LSTMs like ELMo improves results further so they are significantly better than the ELMo+Multi-task ones. We suspect there could be further gains from combining our method with language model pre-training, which we leave for future work.