Multiscale Collaborative Deep Models for Neural Machine Translation
Table 6: Results on CNNDM summarization using ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L). “36L” is short for “36-layer encoder”.
['[BOLD] Model', '[BOLD] R-1', '[BOLD] R-2', '[BOLD] R-L']
Fine-tune BERT for extractive summarization. Lead3 is a baseline which simply selects the first three sentences as the summary. HibertM Zhang et al. PGNet See et al. We also implemented two baselines. One is the standard 6-layer Transformer model. We can see that the deep Msc leads to a +1.8 ROUGE improvement over Transformer. The other baseline is the hierarchical transformer summarization model (HierTrans), which involevs both a sentence-level and a document-level transformer encoders, as well as a standard transformer decoder. Note the settings for both encoders are the same (each of them have L=18, emb=512, ffn=1024, head=8). The deep Msc outperforms HierTrans by 0.5 to 0.7 ROUGE with the same depth of encoders.