Closed-Book Training to Improve Summarization Encoder Memory
Table 6: ROUGE F1 scores of ablation studies, evaluated on CNN/Daily Mail validation set.
['[EMPTY]', 'ROUGE 1', 'ROUGE 2', 'ROUGE L']
Fixed-Encoder Ablation: Next, we conduct an ablation study in order to prove the qualitative superiority of our 2-decoder model’s encoder to the baseline encoder. To do this, we train two pointer-generators with randomly initialized decoders and word embeddings. For the first model, we restore the pre-trained encoder from our pointer-generator baseline; for the second model, we restore the pre-trained encoder from our 2-decoder model. We then fix the encoder’s parameters for both models during the training, only updating the embeddings and decoders with gradient descent. Since these two models have the exact same structure with only the encoders initialized according to different pre-trained models, the significant improvements in metric scores suggest that our 2-decoder model does have a stronger encoder than the pointer-generator baseline. Gradient-Flow-Cut Ablation: We further design another ablation test to identify how the gradients from the closed-book decoder influence the entire model during training. Fig. As we can see, the closed-book decoder only depends on the word embeddings and encoder. Therefore it can affect the entire model during training by influencing either the encoder or the word-embedding matrix. When we stop the gradient flow between the encoder and closed-book decoder (\⃝raisebox{-0.9pt}{1} in Fig. This proves that the gradients back-propagated from closed-book decoder to the encoder can strengthen the entire model, and hence verifies the gradient-flow intuition discussed in introduction (Sec.