A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
Table 5: Manual evaluation results. The scores indicate the percentages of Win, Lose or Tie when our model is compared with a baseline. κ denotes Fleiss’ kappa (all are fair agreement or moderate agreement). The scores marked with * mean p-value< 0.05 and ** indicates p-value< 0.01 in sign test.
['[BOLD] Models', '[BOLD] Grammaticality  [BOLD] Win\xa0(%)', '[BOLD] Grammaticality  [BOLD] Lose\xa0(%)', '[BOLD] Grammaticality  [BOLD] Tie\xa0(%)', '[BOLD] Grammaticality  [ITALIC] κ', '[BOLD] Logicality  [BOLD] Win\xa0(%)', '[BOLD] Logicality  [BOLD] Lose\xa0(%)', '[BOLD] Logicality  [BOLD] Tie\xa0(%)', '[BOLD] Logicality  [ITALIC] κ']
all the results show fair agreement (0.2≤κ≤0.4) or moderate agreement (0.4≤κ≤0.6). We also conducted sign test to check the significance of the differences. The results indicate that our model performs significantly better than other baselines in both metrics. More specifically, post-training on knowledge bases leads to significant improvements in grammar and logic by offering more knowledge for expanding the story plots. And multi-task learning further enhances the performance in logic and does not affect fluency of generated stories.