Heterogeneous Relational Reasoning in Knowledge Graphs with Reinforcement Learning
Table 2: Experimental results on the NELL-995, Amazon Beauty, and Amazon Cellphones data sets. @{1, 3, 5, 10} and MRR are standard KG completion metrics and are described in Section 4.1. The methods are separated into embedding-based (ConvE, ComplEx, and DistMult) and agent-based (MINERVA, Lin et al., and Ours (TN)) groups. Bolded numbers indicate the best-performing method from each group, with respect to the given metric.
['[BOLD] Data Set Metric (%)', '[BOLD] NELL-995 @1', '[BOLD] NELL-995 @5', '[BOLD] NELL-995 @10', '[BOLD] NELL-995 MRR', '[BOLD] Amazon Beauty @1', '[BOLD] Amazon Beauty @5', '[BOLD] Amazon Beauty @10', '[BOLD] Amazon Beauty MRR', '[BOLD] Amazon Cellphones @1', '[BOLD] Amazon Cellphones @5', '[BOLD] Amazon Cellphones @10', '[BOLD] Amazon Cellphones MRR']
Embedding-based methods show an overall higher performance compared to the RL-based methods. We can see that in all three datasets, our results outperform both RL baselines (Lin et al. Amazon datasets, on the other hand, are far more challenging. We notice that even the embedding based methods are struggling with low performance on these datasets. On the Amazon data, the performance of all methods is significantly lower. Our method results in a 4% improvement in MRR (and 5.43% in Hits@1) over the best RL baseline on Amazon Cellphones and a 3.9% improvement in MRR (and 5.5% in Hits@1) on Amazon Beauty. On the NELL-995 dataset, our method results in 2.8% improvement in MRR and 4% improvement in Hits@1 over the best performing baseline. We also performed ablations studies to analyze the effect of each module in our model. We removed the type embeddings in Ours (-T) and the heterogeneous neighbor encoder in Ours (-N). We notice that removing the heterogeneous neighbor encoder results in a higher drop in performance in the Amazon datasets. This gap is quite smaller in the NELL-955 data.