What Happened? Leveraging VerbNet to Predict the Effects of Actions in Procedural Text
Table 3: Results of two (separate) ablation experiments: (a) only use the original VerbNet axioms (ignore extensions/corrections) (b) only use inferences from the Semantic Lexicon (ignore inference rules about projection, colocation, etc.). F1 differences between the full and ablation versions are statistically significant (p<0.05).
['ProComp', 'Single Answer Questions', 'Single Answer Questions', 'Single Answer Questions', 'All Questions', 'All Questions', 'All Questions']
We are also interested in two related questions: How much have our extensions to the original VerbNet-derived lexicon helped? How much has additional commonsense inference, beyond computing direct consequences of actions, helped? To evaluate this, we performed two (independent) ablations: (1) removing the extensions and corrections made to the original VerbNet axioms in the Lexicon (2) disabling the inference rules about change; thus only states that directly follow from the Semantic Lexicon, rather than inferred via projection etc., are recovered. For the first ablation, the results indicate that the VerbNet extensions have significantly improved performance (+5.9% F1 single answer questions, +2.4% F1 all questions). The second ablation similarly demonstrates additional inference using the commonsense rules improves performance (+6.6% F1 single answer questions, +2.1% F1 all questions). The relatively high performance even without this additional inference suggests that many questions in OldProPara ask about direct effects of actions. We analyze these further below.