A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization
Table 2: Comparison of generated (Gen) and missed (Mis) tokens for different training setups. We divide token in goldstandard summaries by 1) if it is seen in abstracts during training, and 2) if it is in the input text.
['[EMPTY]', '[BOLD] Seen in Training (%)  [ITALIC] In Input', '[BOLD] Seen in Training (%)  [ITALIC] In Input', '[BOLD] Seen in Training (%)  [ITALIC] In Input', '[BOLD] Seen in Training (%)  [ITALIC] Not In Input', '[BOLD] Seen in Training (%)  [ITALIC] Not In Input', '[BOLD] Seen in Training (%)  [ITALIC] Not In Input', '[BOLD] Unseen (%)']
We further classify the words in gold-standard summaries based on if they are seen in abstracts during training and then whether they are taken from the input text. We examine whether they are generated correctly. Full training set of opinion is used for in-domain and mix-domain training. Nonetheless, model trained for opinion is better at generating new words not in the input (8.2% vs. 2.6%). This is consistent with our observation that in opinion domain human editors favors new words different from the input.