On the impressive performance of randomly weighted encoders in summarization tasks
Table 1: Results of the CNN / Daily Mail test dataset. ROUGE scores have a 95% confidence interval of at most ±0.24 as reported by the official ROUGE script.
['Model', 'Encoder', 'ROUGE 1', 'ROUGE 2', 'ROUGE L']
Of note, our hierarchical random encoder models (Random H-LSTM and Random LSTM + ESN) obtain ROUGE scores close to other trained models. The gap between our trained and random hierarchical encoders is about 1.1 point for all ROUGE scores. Hierarchical random encoders even outperform a competitive baseline nallapati2016abstractive in terms of ROUGE-2, even though the cited model uses pre-trained embeddings. With respect to the trained H-LSTM, the Random H-LSTM achieves ROUGE scores that are very similar: the gap is 3.5% in ROUGE-1, 8.5% in ROUGE-2 and 3.9% in ROUGE-L. We also tested the Identity H-LSTM to get an idea of the role that trained word embeddings play in the performance. To measure the representational power of random projection in the encoder, we compare the ROUGE scores of random hierarchical encoder models with those of the Identity H-LSTM model. This has brought us closer to gauging the effectiveness of randomly projected recurrent encoder hidden states over an accumulation of word embeddings.