Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language?
Table 3: Results for productivity. 0ptd indicates the set of inferences where the embedding depth is d.
['[BOLD] Train', '[BOLD] Dev/Test', '[BOLD] BERT', '[BOLD] LSTM', '[BOLD] TreeLSTM']
The accuracy on 0pt1 and 0pt2 was nearly 100%, indicating that all models almost completely generalize to inferences containing previously seen depths. When 0pt1+0pt2 were used as the training set, the accuracy of all models on 0pt3 exceeded chance. Similarly, when 0pt1+0pt2+0pt3 were used as the training set, the accuracy of all models on 0pt4 exceeded chance. This indicates that all models partially generalize to inferences containing embedded clauses one level deeper than the training set.