No Training Required: Exploring Random Encoders for Sentence Classification
Table 5: Accuracy on single-sentence binary classification tasks from SentEval, where max-pooling is done over padded values instead of over the length of the sentence. Experiments are split between Sorted where sentences are sorted in order of length prior to batching and Unsorted where they are not.
['[EMPTY]', 'Model', 'MR', 'CR', 'MPQA', 'SUBJ', 'SST2']
The first half of the table shows results when max pooling with padding is used and the batches are sorted. The second half of the table shows performance when the batches are unsorted. As can be seen by the table, the performance has a significant drop-off when the batches are unsorted, especially for MR and CR. Max pooling over padded values changes negative values in the features of longer sentences to zero. This is because if the largest value in the hidden representations over the length of the sentence is negative, the padded zeros will be greater. Thus, longer sentences, when grouped with shorter ones, will have more sparse representations. We tried to reproduce this effect by using a ReLU, but it didn’t increase performance. In fact it is 0.0 for CR, one of the tasks most affected by this phenomenon.