Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization
Table 6: Effect on best-performing GRU of replacing words w by unseen synonyms s(w) in the test set and providing the model with the corresponding GloVe embedding.
['[ITALIC] w', '[ITALIC] s( [ITALIC] w)', '[ITALIC] cos_ [ITALIC] dist( [ITALIC] w, [ITALIC] s( [ITALIC] w))', 'test accuracy containing  [ITALIC] w', 'test accuracy containing  [ITALIC] s( [ITALIC] w)']
Synonyms One of the most basic relations on the level of lexical semantics is synonymy, which holds between words with equivalent meanings. In the language L, a word can be substituted with one of its synonyms without altering the entailment relation assigned to the sentence pairs that contain it. If the GRU manages to perform well on such a modified data set after receiving the pretrained GloVe embedding of the unseen word, this is a first piece of evidence for its zero-shot generalization skills. We test this for several pairs of synonymous words. The best-performing GRU is first evaluated with respect to the fragment of the test data containing the original word w, and consequently with respect to that same fragment after replacing the original word with its synonym s(w). Apparently, the word embeddings of the synonyms encode the lexical properties that the GRU needs to recognize that the same entailment relations apply to the sentence pairs. This does not prove that the model has distilled essential information about hyponymy from the GloVe embeddings. It could also be that the word embeddings of the replacement words are geometrically very similar to the originals, so that it is an algebraic necessity that the same results arise. However, this suspicion is inconsistent with the result of changing ‘hate’ into ‘detest’. The cosine distance between these words is 0.56, so according to this measure their vectors are more similar than those representing ‘love’ and ‘adore’ (which have a cosine distance of 0.57). Nonetheless, replacing ‘hate’ with ‘detest’ confuses the model, whereas substitution of ‘love’ into ‘adore’ only decreases testing accuracy by 4.5 percentage points. This illustrates that robustness of the GRU in this respect is not a matter of simple vector similarity. In those cases where substitution into synonyms does not confuse the model it must have recognized a non-trivial property of the new word embedding that licenses particular inferences.