Compositionality and Generalization in Emergent Languages
Table 1: Feeding shuffled messages from the analyzed language to the corresponding trained Receiver. Average percentage accuracy across 10 random shufflings (s.d. always ≈0) when: top: symbols in all positions but one are shuffled across the data-set; bottom: symbols in a single position are shuffled across the data-set. The data-set includes all training and test messages produced by the trained Sender and correctly decoded in their original form by Receiver (>99% of total messages).
['[ITALIC] fixing', '[ITALIC] pos1', '[ITALIC] att1 1', '[ITALIC] att2 3', '[ITALIC] both atts 0']
Look first at the top block, where the trained Receiver of the relevant run is fed messages with the symbol in one original position preserved, the others shuffled. Confirming that communication is largely happening by disentangled means, preserving pos2 alone suffices to have Receiver guessing a large majority of att2 values, and keeping pos3 unchanged is enough to guess almost 90% of att1 values correctly. Conversely, preserving pos1 alone causes a complete drop in accuracy for both attributes. However, neither pos2 nor pos3 are sufficient on their own to perfectly predict the corresponding attributes. Indeed, the results in the bottom block of the table (one symbol shuffled while the others stay in their original position) confirm that pos1 carries useful complementary information: when fixing the latter and either one of the other positions, we achieve 100% accuracy for the relevant attribute (att2 for pos1+pos2 and att1 for pos1+pos3), respectively.