A Hybrid Neural Network Model for Commonsense Reasoning
Table 4: Test results
['[EMPTY]', 'WNLI', 'WSC', 'PDP60']
Compared with all the baselines, HNN obtains much better performance across three benchmarks. This clearly demonstrates the advantage of the HNN over existing models. For example, HNN outperforms the previous state-of-the-art BERTWiki-WSCR model with a 11.7% absolute improvement (83.6% vs 71.9%) on WNLI and a 2.8% absolute improvement (75.1% vs 72.2%) on WSC in terms of accuracy. Meanwhile, it achieves a 11.7% absolute improvement over the previous state-of-the-art BERTLARGE-LM model on PDP60 in accuracy. Note that both BERTWiki-WSCR and BERTLARGE-LM are using language model-based approaches to solve the pronoun resolution problem. On the other hand, We observe that DSSM without pre-training is comparable to BERTLARGE-LM which is pre-trained on the large scale text corpus (63.0% vs 62.0% on WSC and 75.0% vs 78.3% on PDP60). Our results show that HNN, combining the strengths of both DSSM and BERTWSCR, has consistently achieved new state-of-the-art results on all three tasks. To further boost the WNLI accuracy on the GLUE benchmark leaderboard, we record the model prediction at each epoch, and then produce the final prediction based on the majority voting from the last six model predictions. HNNensemble brings a 5.4% absolute improvement (89.0% vs 83.6%) on WNLI in terms of accuracy.