Posing Fair Generalization Tasks for Natural Language Inference
Table 3: Mean accuracy of 5 runs on our difficult but fair generalization task, with standard 95% confidence intervals. These models are trained on the intermediate predictions described in Section 5.
['Model', 'Train', 'Dev', 'Test']
The four standard neural models fail the task completely. The CompTreeNN and CompTreeNTN, while better, are not able to solve the task perfectly either. However, it should be noted that the CompTreeNN outperforms our four standard neural models by ≈30% and the CompTreeNTN improves on this by another ≈10%. This increase in performance leads us to believe there may be some other composition function that solves this task perfectly.