No Training Required: Exploring Random Encoders for Sentence Classification
Table 1: Performance (accuracy for all tasks except SICK-R and STSB, for which we report Pearson’s r) on all ten downstream tasks where all models have 4096 dimensions with the exception of BOE (300) and ST-LN (4800). Standard deviations are show in parentheses. InferSent-1 is the paper version with GloVe (G) embeddings, InferSent-2 has fixed padding and uses FastText (F) embeddings, and InferSent-3 has fixed padding and uses GloVe embeddings. We also show the difference between the best random architecture (BestRand) and InferSent-3 and ST-LN, respectively. The average performance difference between the best random architecture and InferSent-3 and ST-LN is 1.7 and -0.4 respectively.
['Model', 'Dim', 'MR', 'CR', 'MPQA', 'SUBJ', 'SST2', 'TREC', 'SICK-R', 'SICK-E', 'MRPC', 'STSB']
We hypothesize that SkipThought might do a lot better if it had been trained with better pre-trained word embeddings. If we take Cover’s theorem to the limit, we can project to an even higher-dimensional representation as long as we can still easily fit things onto a modern GPU: In order to make for a fair comparison, we can also randomly project InferSent and SkipThought representations to the same dimensionality and examine performance.