Compositionality and Generalization in Emergent Languages
Table 6: Feeding shuffled messages from the medium-posdis and low-posdis languages to the corresponding trained Receivers. Mean percentage accuracy across 10 random shufflings (standard deviation is always ≈0) when: top: symbols in all positions but one are shuffled across the data-set; middle: symbols in a single position are shuffled across the data-set; bottom: shuffling the symbols within each message (ensuring all symbols move). The data-set includes all training and test messages produced by the trained Sender and correctly decoded in their original form by Receiver (>99% of total messages).
['[EMPTY]', '[EMPTY]', '[ITALIC] medium-posdis  [ITALIC] att1', '[ITALIC] medium-posdis  [ITALIC] att2', '[ITALIC] medium-posdis  [ITALIC] both', '[ITALIC] low-posdis  [ITALIC] att1', '[ITALIC] low-posdis  [ITALIC] att2', '[ITALIC] low-posdis  [ITALIC] both']
The results for medium-posdis are discussed in the main text. We observe here how virtually any ablation strongly impacts accuracy in denoting either attribute by the highly entangled low-posdis language. This points to another possible advantage of (partially) disentangled languages such as medium-posdis: since pos2 and pos3 are referring to att2 and att1 independently, in ablations in which they are untouched, Receiver can still retrieve partial information, by often successfully guessing the attribute they each refer to. We also report in the table the effect of shuffling across the positions of each message. This is very damaging not only for medium-posdis, but for low-posdis as well, showing that even the latter is exploiting positional information, albeit in an inscrutable, highly entangled way. Note in Fig.