HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
Table 2: Results of various models on the NYT50 test set using full-length F1 ROUGE. \sc HibertS (in-domain) and \sc HibertM (in-domain) only uses one pre-training stage on the NYT50 training set.
['Models', 'R-1', 'R-2', 'R-L']
EXTRACTION is a extractive model based hierarchical LSTM and we use the numbers reported by \newcitexu:2019:arxiv. The improvement of \sc HibertM over the baseline without pre-training (HeriTransformer) becomes 2.0 ROUGE. \sc HibertS (in-domain), \sc HibertM (in-domain), \sc HibertS and \sc HibertM all outperform BERT significantly according to the ROUGE script.