Compositional Languages Emerge in a Neural Iterated Learning Model
Table 2: Validation performance under different validation set sizes.
['Valid set size', '0 Train', '0 Valid', '8 Train', '8 Valid', '16 Train', '16 Valid', '32 Train', '32 Valid']
In this last series of experiments, we aim to explore the relationship between topological similarity and the generalisation ability of our neural agents, which can also indirectly reflect the expressivity of a language. We measure this ability by looking at their validation game performance: we restrict the training examples to a limited numbers of objects (i.e., the training set), and look at how good are the agents at playing the game on the others (i.e., the validation set). To illustrate the relationship between ρ and validation performance, we randomly choose Ia∈[60,4000] and Ib∈[5,200] and conduct a series of experiments. Those for which Ia and Ib are not in their optimal range will yield a worse performance on both validation test and topological similarity. This shows the linear correlation between these two metrics, and a significance test confirms it: the correlation coefficient is 0.928, and the associated p-value is 3.8∗10−104. Hence, under various experimental settings, the validation performance and the topological similarity are strongly correlated. The fact that the Alice-reset setting performs better than the Bob-reset setting also matches our analysis well.