Towards Lossless Encoding of Sentences
Table 1: Mean squared error loss of stacked LSTMs and our RAE model for different embedding sizes. All models are trained on the autoencoding task for 20 epochs and models of same embedding size have the same capacity. MSE is computed on the BookCorpus dev set Zhu et al. (2015), between the input GloVe embeddings Pennington et al. (2014) and output embeddings.
['Model', '[ITALIC] demb', 'MSE (dev)']
To demonstrate how good our model is at reconstruction, we trained a stacked LSTM on the same autoencoding task. All LSTMs have two encoder and two decoder layers. The 1024 variant seems to have reached a saturation point, as it performs similarly to the 512 version. All RAEs and LSTMs were trained for 20 epochs and models with same embedding size have the same capacity. The LSTM with an embedding size of 1024 only slightly achieves lower MSE than the RAE with embedding size 300.