Emergence of Communication in an Interactive World with Consistent Speakers
Table 2: Context-independence evaluation on the test set.
['[BOLD] Training setup', '[BOLD] 3C/3N/1M', '[BOLD] 5C/3N/1M', '[BOLD] 8C/3N/1M']
A random speaker that samples an utterance randomly sets a lower-bound on CI(⋅) for reference, and indeed values are close to 0. FixedRand is a consistent speaker that is unambiguous, but is not compositional. Unambiguity increases CI(⋅) slightly, but still the score is low. The CI(⋅) score of CCO is higher, but when we replace the GRU that processes u with a bag-of-words, CI(⋅) improves substantially. In the referential game, language emerged with CCO is substantially more context independent than Obverter and PG, especially with the BOW variant. PG exhibits very low context-independence in both the referential game and when tested on Collect with varying colors. In contrast to CCO, PG encodes the message in much longer messages, which might explain the low CI(⋅) score. We run the agents with 3 colors and 3 numbers, while increasing the number of missions. For a fair CI(⋅) comparison, we calculated the score using only the colors and numbers concepts. Despite our prior expectation, we did not observe improvement in context-independence as the number of missions increases.