Faithful to the Original: Fact Aware Neural Abstractive Summarization
Table 5: Final perplexity on the development set. † indicates the value is cited from the corresponding paper. ABS+, Feats2s and Luong-NMT do not provide this value.
['Model', 'Perplexity']
We can see that our model achieves the lowest perplexity compared against the state-of-the-art systems. It is also noted that, FTSumg largely outperforms FTSumc, which verifies the importance of context selection. Although the focus of our model focuses is to improve faithfulness , the ROUGE scores it receives are also much higher than the other methods. Note that, ABS+ and Feats2s have utilized a series of hand-crafted features, but our model is totally data-driven. Even though, our model surpasses Feats2s by 13% and ABS+ by 56% on ROUGE-2. When fact descriptions are ignored, our model is equivalent to the standard attentional s2s model s2s+att. Therefore, it is safe to conclude that, fact descriptions have significant contribute to the increase of ROUGE scores. It also largely explains why FTSumg is superior to FTSumc. FTSumc treats the source sentence and relations equally, while FTSumg tells the fact descriptions are often more reliable, as discussed in more detail later.