Capacity, Bandwidth, and Compositionality in Emergent Language Learning
Table 1. The base hyperparameter counts used in our experiments for each of models A and B.
['[EMPTY]', 'Model A', 'Model B']
We train two sets of models. We gradually decrease the number of LSTM units from the base model by a factor α∈(0,1]. This is how we control the number of parameters (|θ| and |ϕ|), a factor we hypothesize to influence the resulting compositionality. We obtain seven models from each of these by varying the length of the latent sequence l from {19,20,21,22,23,24,25}. These were chosen because we both wanted to show a range of bits and because we need at least 20 bits to cover the 106 strings in L∗ (⌈log2106⌉=20).