Language as an Abstractionfor Hierarchical Deep Reinforcement Learning
Table 1: Final performance of the low-level policy on different training and test instruction distributions (20 episodes). Language outperforms the non-compositional language representation in both absolute performance and relative generalization gap for every setting. Gap is equal to one minus the ratio of between mean test performance and mean train performance; this quantity can be interpreted as the generalization gap. For instructions with language representation, the generalization gap increases by approximately 59.5% from standard generalization to zero-shot generalization while for non-compositional representation the generalization gap increases by 82.2%
['[EMPTY]', '[BOLD]  Standard train ', '[BOLD]  Standard test ', '[BOLD]  Standard gap ', '[BOLD]  Systematic train ', '[BOLD]  Systematic test ', '[BOLD]  Systematic gap ']
Visual generalization. To evaluate this (i.e. (3)), we design the training and test instruction sets that are systematically distinct. We evaluate the agent’s ability to perform such generalization by splitting the 600 instruction sets through the following procedure: (i) standard: random 70/30 split of the instruction set; (ii) systematic : the training set only consists of instructions that do not contain the words red in the first half of the instructions and the test set contains only those that have red in the first half of the instructions. We emphasize that the agent has never seen the words red in the first part of the sentence in training; in other words, the task is zero-shot as the training set and the test set are disjoint (i.e. the distributions do not share support). From a pure statistical learning theoretical perspective, the agent should not do better than chance on such a test set. This suggests that the agent recognizes the compositional structure of the language, and achieves systematic generalization through such understanding. To faithfully evaluate the importance of compositionality, we want a representation that carries the identical information as the language instruction but without the explicit compositional property (but perhaps still to some degree compositional). To this end, we use a Seq2Seq autoencoder with 64 hidden units to compress the 600 instructions into real-valued continuous vectors. The original tokens are fully recovered which indicates that the compression is lossless and the latent’s information content is the same as the original instruction. This embedding is used in place of the GRU instruction embedding. We also observed that adding regularization to the autoencoder decreases the performance of the resulting representation. For example, decreasing the bottleneck size leads to worse performance, as does adding dropout. As shown in the experiments, the performance without dropout is better than the one with. We hypothesize that adding regularization decreases the compositionality of the representation.