A Hybrid Neural Network Model for Commonsense Reasoning
Table 6: Ablation study of the ranking loss. Note that WNLI and WSCR are reported on dev sets while WSC and PDP60 are reported on test sets.
['[EMPTY]', 'WNLI', 'WSCR', 'WSC', 'PDP60']
As in Eqn. The first two are based on the two component models, respectively, and the third one, as defined in Eqn. At first glance, the ranking loss seems redundant. Thus, we compare two versions of HNN trained with and without the ranking loss. We see that without the ranking loss, the performance of HNN drops on three datasets: WNLI, WSCR and WSC. On the PDP60 dataset, without the ranking loss, the model performs slightly better. However, since the test set of PDP60 includes only 60 samples, the difference is not statistically significant. Thus, we decide to always include the ranking loss in the training objective of HNN.