Compose Like Humans: Jointly Improving the Coherence and Novelty for Modern Chinese Poetry Generation
TABLE II: Automatic and human evaluation results. The last line is the results of human-written poems in test set and human evaluation set. All significance tests are measured by t-test, and the results show that the improvements of our model are significant on the dataset, i.e., p-value < 0.01.
['[EMPTY]', '[BOLD] Automatic Evaluation  [BOLD] PPL', '[BOLD] Automatic Evaluation  [BOLD] Rough-L', '[BOLD] Automatic Evaluation  [BOLD] Distinct-1', '[BOLD] Automatic Evaluation  [BOLD] Distinct-2', '[BOLD] Automatic Evaluation  [BOLD] Novelty-2', '[BOLD] Automatic Evaluation  [BOLD] Novelty-3', '[BOLD] Human Evaluation  [BOLD] Fluency', '[BOLD] Human Evaluation  [BOLD] Coherence', '[BOLD] Human Evaluation  [BOLD] Impressiveness', '[BOLD] Human Evaluation  [BOLD] Poeticness']
Our proposed method GRR outperforms other models almost on all metrics. GRR-Refine receives the lowest PPL values, which shows that bidirectional sentence-level context is beneficial to fluency. It has been proven that lower PPL values usually correspond to simple and general sentences, thus a higher PPL for GRR indicates that our model can generates more diverse and novel words. A higher Rough-L score shows that our generated sentences are more similar to their references, which also implies their diversity. GRR receives the best evaluation on all metrics. The evaluation results on Coherence and Impressiveness prove that our model can jointly improve coherence, diversity and novelty. Compared with DN and WM, GRR-Refine has higher Coherence score, which indicates the validity of bidirectional sentence-level context for improving the coherence. Besides, human evaluation results on Fluency and Coherence show that PPL is not totally reliable for sentence fluency and especially poem coherence. For human, a sentence with diverse language usage can also be fluent but its PPL value may be high.