HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
Table 1: Results of various models on the CNNDM test set using full-length F1 Rouge-1 (R-1), Rouge-2 (R-2), and Rouge-L (R-L).
['Model', 'R-1', 'R-2', 'R-L']
Pointer+Coverage See et al. SentRewrite Hsu et al. all try to decompose the word by word summary generation into sentence selection from document and “sentence” level summarization (or compression). Bottom-Up Gehrmann et al. The extractive models are usually based on hierarchical encoders (SummaRuNNer; Nallapati et al. They have been extended with reinforcement learning (Refresh; Narayan et al. Zhang et al. Lead3 is a baseline which simply selects the first three sentences. Our model \sc HibertS (in-domain), which only use one pre-training stage on the in-domain CNNDM training set, outperforms all of them and differences between them are all significant with a 0.95 confidence interval (estimated with the ROUGE script). Note that pre-training \sc HibertS (in-domain) is very fast and it only takes around 30 minutes for one epoch on the CNNDM training set. We also implemented two baselines. Note the setting for HeriTransfomer is (L=4,H=300 and A=4) Another baseline is based on a pre-trained BERT Devlin et al. We used the BERTbase model because our 16G RAM V100 GPU cannot fit BERTlarge for the summarization task even with batch size of 1. The positional embedding of BERT supports input length up to 512 words We feed each block (the BOS and EOS tokens of each sentence are replaced with [CLS] and [SEP] tokens) into BERT and use the representation at [CLS] token to classify each sentence. Our model \sc HibertS outperforms BERT by 0.4 to 0.5 ROUGE despite with only half the number of model parameters (\sc HibertS 54.6 M v.s. BERT 110M).