On the Effective Use of Pretraining for Natural Language Inference
Table 1: Accuracy results for the SNLI dev-set experiment.
['[EMPTY]', 'Gaussian', 'Orthonormal']
After a single epoch, all kinds of pretraining outperform random vectors. Surprisingly, though, GloVe and word2vec also outperform their retrofitted counterparts. In addition, for all but random and word2vec, orthogonal initialization is notably better than Gaussian, and results in a mean gain of 0.8% across all the pretrained embeddings, with gains of 1.0% for GloVe vectors and 2.9% for retrofitted Glove.