Abstractive Summarization for Low Resource Data using Domain Transfer and Data Synthesis
Table 4: ROUGE results. Italics indicates outperforms baselines. Boldface indicates best result over all models. Underlining indicates best result within model group (i.e., Extractive Baselines, PG-net).
['[BOLD] Summarization Model', '[BOLD] Summarization Model', '[BOLD] R-1  [BOLD] CS', '[BOLD] R-2  [BOLD] CS', '[BOLD] R-L  [BOLD] CS', '[BOLD] R-1  [BOLD] ENGR', '[BOLD] R-2  [BOLD] ENGR', '[BOLD] R-L  [BOLD] ENGR', '[BOLD] Row']
The motivation for using domain transfer and data synthesis is our hypothesis (H1). All ROUGE scores for PG-net that outperform all extractive baselines (in italics) involve tuning and/or use of synthesised data, except for one R-1 (row 18). For PG-net, comparing the CNN/DM out-of-domain and Student Reflection in-domain results in rows (5 and 6) and (17 and 18) with their corresponding tuned results in rows 9 and 21, we see that fine tuning improves R-1, R-2, and R-L for all courses (rows 5, 6, 9 and 17, 18, 21). Over all courses, the tuned version of PG-net consistently outperforms the best baseline result for each metric (rows 9 vs. 1, 2, 3, 4 and 21 vs. 13, 14, 15, 16) except for R-2 in Stat2016. To validate our next set of hypothesises (H3, H4. H5), we use the synthesized data in two settings: either using it for training (rows 7, 8 and 19, 20) or tuning (rows 10, 11 and 22, 23). It also shows that while adding synthetic data from the baseline is not always helpful, adding synthetic data from the template model helps to improve both the training and the tuning process. In both CS and ENGR courses, tuning with synthetic data enhances all ROUGE scores compared to tuning with only the original data. (rows 9 and 11). As for Stat2015, R-1 and R-L improved, while R-2 decreased. For Stat2016, R-2 and R-L improved, and R-1 decreased (rows 21 and 23). Training with both student reflection data and synthetic data compared to training with only student reflection data yields similar improvements, supporting H3 (rows 6, 8 and 18, 20). While the increase in ROUGE scores is small, our results show that enriching training data with synthetic data can benefit both the training and tuning of other models. In general, the best results are obtained when using data synthesis for both training and tuning (rows 11 and 23), supporting H5.