Counterfactual Story Reasoning and Generation
Table 5: Pairwise human comparison between the best model (GPT2-M + Sup) and comparison models on all three questions. “Neutral” means both are “equally good”. Percentage of “equally bad” are omitted.
['PLOT - Human Judges Preferred Best model', 'PLOT - Human Judges Preferred Best model', 'PLOT - Human Judges Preferred Neutral', 'PLOT - Human Judges Preferred Comparator', 'PLOT - Human Judges Preferred Comparator']
, we present the human preference results, showing that the best model outperforms the comparison baselines in terms of consistency with premise, while being less consistently better with regards to the other two questions. Interestingly, a model that performs better on one of the evaluated dimensions often performs worse for another question, indicating plenty of room for future work in counterfactual reasoning for story rewriting.