Learning by Semantic Similarity Makes Abstractive Summarization Better
Table 2: Human Evaluation score of the Systems
['[EMPTY]', 'Creativity', 'Readability', 'Relevance', 'Total (Avg)']
Results The table shows averaged score across all 475 responses (5 examples X (95 people). We convert the human evaluation scores from 1-4 scale to 0-100 scale and reported total score, which is the average of three scores. We compare the scores of our model, the baseline, and the reference summary. In terms of the total score, generated summaries from our model scored better than BART (p-value < 0.0061) and reference summaries (p-value <10−10) with statistical significance. In detail, our model performed better than BART on two criteria, namely Creativity (p-value < 0.0037) and Relevance (p-value < 0.0084), and outperformed reference summaries on all three criteria. BART was better than reference summaries in terms of Readability (p-value <10−10) and Relevance (p-value <10−10), but not for Creativity. We calculated p-values using the one-tailed t-test (statistical significance of 0.01). In general, we assume that the test set and training set are of the same quality because the test set intrinsically has a very similar distribution to the training set. If the test set is a low-quality data, then the training set, which is used for training, would be low quality as well. And this low quality will be reflected eventually when we train a model. Hence, it is rare for the model to produce results that are better than the dataset. However, our model has shown contradicting results: our model generated summaries are evaluated as better than the reference summaries. We believe that this is due to the transfer learning of large-scale language models. Language models are pre-trained on various corpus and we exploited this advantage. We assume that the performance difference comes from the language model. The idea of the language model is to learn the general understanding of a language during the pre-training process and the specific task during the fine-tuning process. BART, the underlying structure of our model, is trained to extract salient information by using the encoder and to make a complete sentence by using the decoder. Generating a complete sentence is closely related to providing a readable sentence and this can be achieved by extracting salient information, which can enhance relevance. The successful use of pre-trained knowledge is the main reason why our model and BART received favorable scores. In addition, by considering that the human evaluation score is subjective, the Readability score of 80 is regarded as a high score.