Improving Abstractive Text Summarization with History Aggregation
Table 4: The aggregation mechanism experiments. our experiments use 3 aggregation methods with 2 different aggregation layers.
['Model', 'ROUGE-1', 'ROUGE-2', 'ROUGE-L']
The exploration includes our baseline model(m1) and Transformer model with add function(m2), projection aggregation method(m4) and attention aggregation method(m6). There are 3 groups of experiments with different number of aggregation layers: Transformer adding last 2 layers(m2) and last 3 layers(m3), Transformer with projection aggregation method using 1 layer(m4) and 2 layers(m5) and Transformer with attention aggregation method using 1 layer(m6) and 2 layer(m7). For all models except the exploration of encoder/decoder layers, we use 4 encoder and 4 decoder layers. Aggregation mechanism Analysis If we use baseline model adding the last L layer(s) simply(m2), the result scores will decrease beyond our expectation. However, simply adding the last L layer(s) can re-distribute the encoder final states with history states, it will average the importance weights of those layers and that maybe get things worse. Compared with the baseline model, the result scores of our aggregation models(m4, m6) are boosting. We compute attention between history(query) and encoder final states(key/value) to re-distribute the final states so that the encoder obtains the ability to fusing history information with different importance.