Contextual Graph Attention for Answering Logical Queries over Incomplete Knowledge Graphs
Table 2. Macro-average AUC and APR over test queries with different DAG structures are used to evaluate the performance. All and H-Neg. denote macro-averaged across all query types and query types with hard negative sampling (see Section 3.2.3).
['Dataset Metric', 'Bio AUC', 'Bio AUC', 'Bio APR', 'Bio APR', 'DB18 AUC', 'DB18 AUC', 'DB18 APR', 'DB18 APR', 'WikiGeo19 AUC', 'WikiGeo19 AUC', 'WikiGeo19 APR', 'WikiGeo19 APR']
We use the ROC AUC score and average percentile rank (APR) as two evaluation metrics. All evaluation results are macro-averaged across queries with different DAG structures The hyper-parameters for the baseline models GQE are tuned using grid search and the best ones are selected. Then we follow the practice of Hamilton et al. We use Adam optimizer for model optimization. The overall delta of CGA over GQE reported in Tab. This is because CGA will significantly outperform GQE in query types with intersection structures, e.g., the 9th query type in Fig. Macro-average computation over all query types makes the improvement less obvious. In order to compare the performance of different models on different query structures (different query types), we show the individual AUC and APR scores on each query type in three datasets for all models ( To highlight the difference, we subtract the minimum score from the other scores in each figure. We can see that our model consistently outperforms the baseline models in almost all query types on all datasets except for the sixth and tenth query type (see In both these two query types, GQE+KG[min] has the best performance. The advantage of our attention-based models is more obvious for query types with hard negative sampling strategy. For example, as for the 9th query type (Hard-3-inter) in Fig. Note that this query type has the largest number of neighboring nodes (3 nodes) which shows that our attention mechanism becomes more effective when a query type contains more neighboring nodes in an intersection structure. This indicates that the attention mechanism as well as the original KG training phase are effective in discriminating the correct answer from misleading answers.