Generating Abstractive Summaries from Meeting Transcripts
Table 7: Readability estimates of summaries.
['[BOLD] Method', '[BOLD] Readability score', '[BOLD] Log likelihood']
To evaluate the linguistic quality of the generated summaries, we perform readability analysis. We asked one human judge to mark sentences in the generated abstractive summaries as either readable or not readable. Readability indicates how well the idea in the sentence is conveyed to the reader. Excessive presence of disfluencies or ill-formed utterances should be marked as not readable. We provided these instructions to the human judge. Out of 261 summary sentences generated for the 20 abstractive summaries, 67 sentences were found to be not readable (∼26%). We also manually evaluated the extracted utterances and found that 33% of the utterances contained various disfluencies, making them difficult to read. We also performed another readability analysis for the summaries generated using MSC and found that only 62% of the generated sentences in the summaries are readable. The readability of MSC summaries (0.62) is even worse than that of the extractive summaries (0.67), showing that, while the generated sentences in MSC model are informative (high ROUGE scores), they suffer from serious grammatical issues as no factor of linguistic quality is considered in the model. Generally, several utterances in extractive summaries were marked as not readable due to excessive use of disfluencies. Furthermore, there were incomplete utterances that created confusion in the minds of the reader. For example, an extracted utterances – “Ah eagle , right okay .” – although grammatical, does not tell us anything about what is being spoken about. To obtain a coarse estimate of grammaticality, we also calculate the average log-likelihood score provided by the Stanford Parser. We compute the average log-likelihood scores of the confidence of the dependency parses for each type of summaries.