Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization
Table 7: Effect on best-performing GRU of replacing words w by unseen ontological twins t(w) in the test set and providing the model with the corresponding GloVe embedding.
['[ITALIC] w', '[ITALIC] t( [ITALIC] w)', '[ITALIC] cos_ [ITALIC] dist( [ITALIC] w, [ITALIC] t( [ITALIC] w))', 'test accuracy containing  [ITALIC] w', 'test accuracy containing  [ITALIC] t( [ITALIC] w)']
Examples of ontological twins in the taxonomy of nouns NL are ‘Romans’ and ‘Venetians’ . The same holds for e.g. ‘Germans’ and ‘Polish’ or for ‘children’ and ‘students’. For several such word-twin pairs the GRU is evaluated with respect to the fragment of the test data containing the original word w, and with respect to that same fragment after replacing the original word with ontological twin t(w). Replacing ‘Romans’ with other urban Italian demonyms hardly affects model accuracy on the modified fragment of the test data. As before, there appears to be no correlation with vector similarity because the cosine distance between the different twin pairs has a much higher variation than the corresponding accuracy scores. ‘Germans’ can be changed into ‘Polish’ without significant deterioration, but substitution with ‘Dutch’ greatly decreases testing accuracy. The situation is even worse for ‘Spanish’. Again, cosine similarity provides no explanation - ‘Spanish’ is still closer to ‘Germans’ than ‘Neapolitans’ to ‘Romans’. Rather, the accuracy appears to be negatively correlated with the geographical distance between the national demonyms. After replacing ‘children’ with ‘students’, ‘women’ or ‘linguists’, testing scores are still decent.