Learning Dynamic Belief Graphs to Generalizeon Text-Based Games
Table 2: Agents’ normalized test scores and averaged relative improvement (%↑) over Tr-DQN across difficulty levels. An agent m’s relative improvement over Tr-DQN is defined as (Rm−RTr-DQN)/RTr-DQN where R is the score. All numbers are percentages. \definecolor[named]pgfstrokecolorrgb0.953125,0.56640625,0.359375\pgfsys@color@rgb@stroke0.9531250.566406250.359375\pgfsys@color@rgb@fill0.9531250.566406250.359375\vardiamondsuitrepresents ground-truth full graph; \definecolor[named]pgfstrokecolorrgb0.0078125,0.48828125,0.7109375\pgfsys@color@rgb@stroke0.00781250.488281250.7109375\pgfsys@color@rgb@fill0.00781250.488281250.7109375♣represents discrete Gt generated by GATA-GTP; \definecolor[named]pgfstrokecolorrgb0.04296875,0.328125,0\pgfsys@color@rgb@stroke0.042968750.3281250\pgfsys@color@rgb@fill0.042968750.3281250♠represents Ot. ⋆and ∞are continuous Gt generated by GATA, when the graph updater is pre-trained with OG and COC tasks, respectively.
['Difficulty Level', '20 Training Games 1', '20 Training Games 2', '20 Training Games 3', '20 Training Games 4', '20 Training Games 5', '20 Training Games %↑', '100 Training Games 1', '100 Training Games 2', '100 Training Games 3', '100 Training Games 4', '100 Training Games 5', '100 Training Games %↑', 'Avg. %↑']
Equipped with belief graphs, GATA significantly outperforms all text-based baselines. The graph updater pre-trained on both of the self-supervised tasks We observe further improvements in GATA’s policies when the text observations (\definecolor[named]pgfstrokecolorrgb0.04296875,0.328125,0\pgfsys@color@rgb@stroke0.042968750.3281250\pgfsys@color@rgb@fill0.042968750.3281250♠) are also available. We believe the text observations guide GATA’s action scorer to focus on currently observable objects through the bi-attention mechanism. The attention may further help GATA to counteract accumulated errors from the belief graphs. In addition, we observe that Tr-DRQN and Tr-DRQN+ outperform Tr-DQN, with 3.9% and 3.6% relative improvement (%↑). This suggests the implicit memory of the recurrent components improves performance. We also observe GATA substantially outperforms Tr-DQN when trained on 100 games, whereas the DRQN agents struggle to optimize rewards on the larger training sets. Consistent with GATA, we find GATA-GTP also performs better when given text observations (\definecolor[named]pgfstrokecolorrgb0.04296875,0.328125,0\pgfsys@color@rgb@stroke0.042968750.3281250\pgfsys@color@rgb@fill0.042968750.3281250♠) as additional input to the action scorer. Although GATA-GTP outperforms Tr-DQN by 14.9% when text observations are available, its overall performance is still substantially poorer than GATA. Although the graph updater in GATA-GTP is trained with ground-truth graphs, we believe the discrete belief graphs and the discrete operations for updating them (Appx. We also show the agents’ training curves and examples of the belief graphs G generated by GATA. Figure We observe consistent trends with the testing results of (Right) highlights the sparsity of a ground-truth graph compared to that of a belief graph G. Since generation of G is unsupervised by any ground-truth graphs, we do not expect G to be interpretable nor sparse. Further, since the self-supervised models learn belief graphs directly from text, some of the learned features may correspond to the underlying grammar or other features useful for the self-supervised tasks, rather than only being indicative of relationships between objects. However, we show G encodes useful information for a relation prediction probing task in Appx.