Biases for Emergent Communication inMulti-agent Reinforcement Learning
Table 2: Proportion and average reward of good runs. Values are means over 50 runs with 95% confidence intervals, calculated using Wilson approximation in the case of Bernoulli variables.
['Biases', 'Proportion good', 'CI', 'Final reward (good runs)', 'Final reward']
We find that biases for positive signalling and positive listening both lead to increased reward, and adding either bias to the agents leads to more consistent discovery of useful communication protocols; we define these as runs which get reward greater than 13, the maximum final reward in 50 runs with no communication. With or without biases, the agents still frequently only discover local optima - for example, protocols where the agent which can find treasure reports on the status of only one tunnel, leaving the other to search the remaining tunnels. This demonstrates a limitation of these methods; positive signalling and listening biases are useful for finding some helpful communication protocol, but they do not completely solve the joint exploration problem in emergent communication. However, among runs which achieve some communication, we see greater reward on average among runs with both biases, corresponding to reaching better local optima for the communication protocol on average. Next, we intervene in the environment to demonstrate that these communication protocols have the expected effect on the collector. For each of these pairs of agents, we produce a version of the environment where the message channel is overridden, starting after 100 frames. We override the channel with a constant message, using the symbol which most strongly indicates a particular tunnel. We then measure how long the collector takes to reach a square 3 from the bottom, where the agent is just out of view of the treasure. In both cases, the collector reaches the tunnel significantly faster than in the baseline, indicating that the finder’s consistent communication is being acted on as expected.