The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approach
Table 1: Average percentage of completed games for Home World (± 95% confidence intervals). Q-learning baseline policy was trained once until reaching 15% game completion on average.
['[BOLD] Policy', '[BOLD] Number of Clusters (i.e. | [ITALIC] Z|)  [BOLD] 4', '[BOLD] Number of Clusters (i.e. | [ITALIC] Z|)  [BOLD] 16', '[BOLD] Number of Clusters (i.e. | [ITALIC] Z|)  [BOLD] 24', '[BOLD] Number of Clusters (i.e. | [ITALIC] Z|)  [BOLD] 32']
We evaluate the policies based on the percentage of games completed. It is important to note that since our goal is to evaluate sample efficiency, our policies have been trained on far fewer episodes compared to Narasimhan et al. Further, we have retained the baseline policy’s hyper-parameters for the Bottleneck Simulator policies while reporting the results. We observe peak performance at 36.8% for the Bottleneck Simulator policy with a cluster size of 24, which is significantly higher than the State Abstraction policy at 26.5% and the baseline policy at only 15%. This shows empirically that the Bottleneck Simulator policy is the most sample efficient algorithm, since all policies have been trained on the same number of examples. Finally, it should be noted that the State Abstraction and Bottleneck Simulator policies are complementary and could potentially be combined (e.g. by training a State Abstraction policy from samples generated by the Bottleneck Simulator environment model).