Abstractive Text Summarization based onLanguage Model Conditioning and Locality Modeling
Table 3: Ablation study of the BERT-based model on the truncated and original SwissText dataset
['Model', 'ROUGE-1', 'ROUGE-2', 'ROUGE-L']
On SwissText data we use the multilingual version of BERT-Base. The introduction of BERT into the transformer increased the ROUGE-1, ROUGE-2 and ROUGE-L scores by 7.21, 8.91 and 7.51 percent, respectively. At the same time, the usage of BERT in the decoder decreased the overall score. We assume that the reason behind this is that in multilingual BERT, due to its language-independence, the embedding matrix outputs less precise contextualized representations which undermines their benefits for the summarization task.