Narrative Modeling with Memory Chains and Semantic Supervision
Table 3: Ablation study. Average accuracy over 5 runs on the Story Cloze test set (all subject to the same model selection criterion as our model). Bold: best performance. SD: standard deviation.
['Model', 'Acc.', 'SD']
The uni-directional variant, with its performance down to 77.8, is inferior to its bi-directional cousin. Removing all semantic supervision (resulting in 4 free memory chains) is also damaging to the accuracy (dropping to 77.6). Among the different types of semantic supervision, we see various degrees of performance degradation, with removing sentiment trajectory being the most detrimental, reflecting its value to the task. Interestingly, we observe improvement when removing event sequence supervision from consideration. We suspect that this is mainly due to the noise introduced by the rather inaccurate FrameNet representation output of SEMAFOR (F1=61.4% in frame identification as reported in \newciteDas+:2010). While it is possible that replacing SEMAFOR with the SemLM approach (with the extended frame definition to include explicit discourse markers, e.g. but) in \newcitePeng+:2016 and \newciteChaturvedi+:2017 may potentially reduce the amount of noise , we leave this exercise for future work.