On the impressive performance of randomly weighted encoders in summarization tasks
Table 2: Test perplexity of trained and random hierarchical encoder models on the CNN / Daily Mail dataset (lower is better). Note that Enc = encoder hidden size and Dec = decoder hidden size. Percentages in parentheses are the relative perplexity degradation in random encoder models with respect to the associated Trained H-LSTM encoder model.
['Enc', 'Dec', 'Trained', 'Random', 'Random LSTM']
We chose to base our analysis on perplexity instead of ROUGE to isolate model performance from the effect of word sampling methods such as beam search and to show the quality of overall predicted word probabilities. It is shown that increased model capacity leads to lower test perplexity, which implies better target word prediction. The improvement in performance, however, is not equal across models. We notice that random encoders close the performance gap with the fully trained counterpart as the encoder hidden size increases. For instance, as we vary the encoder hidden size of the Random H-LSTM from 64 to 1024, the relative perplexity gap with the Trained H-LSTM diminishes from 45% to 18%. This pattern aligns with the previous work from notraining, where authors discovered that the performance of random sentence encoder converged to that of trained one as the dimensionality of sentence embeddings increased. We perform similar experiments with the decoder hidden size which varies from 64 to 1024, while fixing the encoder hidden size to 256. We first expected that the hidden size of a trained decoder would play a larger role in enhancing the model performance than that of a random encoder. Moreover, the performance gaps of the previously mentioned configuration with respect to its fully trained counterpart are just as small (19% vs 18% and 20% vs 19%). There are two conclusions that we can draw from this result. First, increasing the capacity of random encoder closes the performance gap between fully trained and random encoder models. Second, increasing the number of parameters in a random encoder yields similar improvements to increasing the number of parameters of a trainable decoder. This illustrates important advantages of using random encoder in terms of the number of parameters to train.