A Study of State Aliasing in Structured Prediction with RNNs
Figure 11: Results on the TextWorld game.
['[ITALIC] λe', '[ITALIC] λv', 'Mean Score (Number of failures)']
Each episode starts with the agent in the bedroom with an empty inventory and terminates after 11 steps. We reward the agent with score values from the game engine. The score goes up by 1 whenever the agent takes an action that moves it closer to the goal, and down by 1 when the agent takes an action that moves it farther away. When the score does not increment (the agent takes a neutral action, such as analyzing an object) , we assign -1. The value estimation, entropy, and policy objectives, respectively, are computed over each training episode as follows: Lv=1ns∑t⎛⎝||∑t′≥tRt′−^Vθ(st)||2⎞⎠Le=1ns∑t(−∑aπθ(st,a)logπθ(st,a))Lp=1ns∑st,at⎛⎝−logπθ(st,at)⎛⎝∑t′≥tRt′−^Vθ(st)⎞⎠⎞⎠, (1) where ns is the number of steps in the episode. We weight and combine these losses as follows to arrive at our overall training objective: L=Lp+λvLv−λeLe. (2) We ran 50 distinct runs of 500 training episodes. These experiments confirm that the best performance results when using both entropy-based exploration and a baseline function with shared parameters. In this case as well, exploration seems to bring the greatest improvement over the basic REINFORCE algorithm. We observe interesting state aliasing phenomena. We can see that these two quantities are coupled: when the Euclidean distance starts to decrease, the probability of going east goes up. This is explained by the fact that after the first time the agent must go west, it must take a key and then go east. When it goes west again, there is no key to take but the option of going east exists. Interestingly, the agent learns to unlock the red chest and open it after going west for the second time because this sequence of actions happens after going west in both cases. However, the last action of inserting the shampoo into the chest is distinct, and the agent’s policy rapidly peaks towards an action that it has seen previously after going west – in this case, going east. Because of the state aliasing, the probability of going east is high; substantial exploration is needed to learn that going east is not optimal and that the two hidden states should be distinct.