Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language?
Table 5: Results for productivity where models were trained with our synthesized dataset mixed with MultiNLI (MNLI).
['[BOLD] Train', '[BOLD] Dev/Test', '[BOLD] BERT', '[BOLD] LSTM', '[BOLD] TreeLSTM']
(46.9 to 100.0), 0pt2 (46.2 to 100.0), and 0pt3 (46.8 to 67.8) while preserving accuracy on MultiNLI (84.6 to 84.4). This indicates that training BERT with our synthetic data does not degrade performance on commonly used corpora like MultiNLI while improving the performance on monotonicity, which suggests that our data-synthesis approach can be combined with naturalistic datasets. For TreeLSTM and LSTM, however, adding our synthetic dataset decreases accuracy on MultiNLI. One possible reason for this is that a pre-training based model like BERT can mitigate catastrophic forgetting in various types of datasets.