An efficient framework for learning sentence representations
Table 1: Comparison of sentence representations on downstream tasks. The baseline methods are GloVe bag-of-words representation, De-noising auto-encoders and FastSent from Hill et al. (2016), the paragraph vector distributed memory model (Le & Mikolov, 2014), skip-thought vectors (Kiros et al., 2015) and the CNN model of Gan et al. (2016). Training times indicated using * refers to CPU trained models and † assumes concatenated representations are trained independently. Performance figures for SDAE, FastSent and ParagraphVec were obtained from Hill et al. (2016). Higher numbers are better in all columns except for the last (MSE). The table is divided into different sections. The bold-face numbers indicate the best performance values among models in the current and all previous sections. Best overall values in each column are underlined.
['[BOLD] Model', '[BOLD] Dim', '[BOLD] Training  [BOLD] time (h)', '[BOLD] MR', '[BOLD] CR', '[BOLD] SUBJ', '[BOLD] MPQA', '[BOLD] TREC', '[BOLD] MSRP  [BOLD] (Acc)', '[BOLD] MSRP  [BOLD] (F1)', '[BOLD] SICK  [BOLD] r', '[BOLD] SICK  [ITALIC] ρ', '[BOLD] SICK  [BOLD] MSE']
The dimensionality of sentence representations and training time are also indicated. For our RNN based encoder we consider variations that are analogous to the skip-thought model. The uni-QT model uses uni-directional RNNs as the sentence encoders f and g. In the bi-QT model, the concatenation of the final hidden states of two RNNs represent f and g, each processing the sentence in a different (forward/backward) direction. The combine-QT model concatenates the representations (at test time) learned by the uni-QT and bi-QT models. We consider two types of encoders trained using our objective - RNN encoders and BoW encoders. Models were trained from scratch on the BookCorpus data. We describe the BoW encoder training below.