Discourse-Based Objectivesfor Fast Unsupervised Sentence Representation Learning
Table 4: Intrinsic evaluation results.
['[EMPTY]', 'conjunction', 'order', 'next']
encoder architecture and whether we train one model per task or one joint model. As expected, the more complex bidirectional GRU architecture is required to capture the appropriate sentence properties, although CBOW still manages to beat the simple GRU (the slowest model), likely by virtue of its substantially faster speed, and correspondingly greater number of training epochs. Joint training does appear to be effective, as both the order and next tasks benefit from the information provided by conjunction. Early experiments on the external evaluation also show that the joint BiGRU model substantially outperforms each single model.