Counterfactual Story Reasoning and Generation
Table 7: Results on automatic metrics for the cross-product of the models and loss functions proposed in Section 4. Bolded results are closest to the human score.
['[EMPTY]', 'BLEU-4', 'ROUGE-L', 'BERT', 'BERT-FT', 'WMS', 'W+SMS']
As expected, the automatic metrics are decently correlated with human scores for adherence to the premise sentence and plot. However, these same metrics correlate negatively with question (3) – adherence to the counterfactual sentence – indicating poor measurement of counterfactual understanding if they were to be reported in their typical manner (i.e., higher score indicating superior performance). Only the BERTScore metrics appear to positively correlate with human scores for counterfactual understanding, making them usable for evaluating generations across properties related to all three questions.