A Hybrid Neural Network Model for Commonsense Reasoning
Table 5: Ablation study of the two component model in HNN. Note that WNLI and WSCR are reported on dev sets while WSC and PDP60 are reported on test sets.
['[EMPTY]', 'WNLI', 'WSCR', 'WSC', 'PDP60']
To answer this question, we first remove each component model, either SSM or MLM, and then report the performance impact of these component models. It is expected that the removal of either component model results in a significant performance drop. For example, with the removal of SSM, the performance of HNN is downgraded from 77.1% to 74.5% on WNLI. Similarly, with the removal of MLM, HNN only obtains 75.1%, which amounts to a 2% drop. All these observations clearly demonstrate that SSM and MLM are complementary to each other and the HNN model benefits from the combination of both.