Counterfactual Story Reasoning and Generation
Table 4: Likert scale scores for different models. The top performing model for each question is bolded.
['Model', '[BOLD] Pre (1)', '[BOLD] Plot (2)', '[BOLD] CF (3)']
Setup In this setting, workers from Amazon Mechanical Turk were presented 100 outputs from 14 different models. For each example, two workers were presented the original premise sentence, the original ending, the counterfactual sentence, and the rewritten ending, and asked to answer the following three questions on a 3-point Likert scale: (1) Does the rewritten ending keep in mind details of the original premise sentence? (2) Is the plot of the rewritten ending relevant to the plot of the original ending? (3) Does the rewritten ending respect the changes induced by the counterfactual sentence? In addition to evaluating the 14 models, we also provided gold human annotated counterfactual endings for the same 100 test examples to compute an expected upper bound for how models should perform. Domain Adaptation helps the model learn to produce relevant rewrites for counterfactuals, especially for models with fewer parameters. Interestingly, however, fine-tuning with the larger set of counterfactuals (CF loss) does not seem to help in rewriting endings that relate to the counterfactuals well.