No Training Required: Exploring Random Encoders for Sentence Classification
Table 3: Performance on a set of probing tasks defined in (Conneau et al., 2018). All random architecture models are 4096 dimensions and were selected by tuning over validation performance on the classification tasks.
['Model', 'SentLen', 'WC', 'TreeDepth', 'TopConst', 'BShift', 'Tense', 'SubjNum', 'ObjNum', 'SOMO', 'CoordInv']
From the table, we see that ESNs and RandLSTMs outperform BOE and BOREP on most of the tasks, especially those that require knowledge of the order of the words. This indicates that these models, even though initialized randomly, are capturing order, as one would expect. We also see that ESNs and InferSent are fairly close on many of the tasks, with Skipthought-LN generally outperforming both.