An Open-World Extension to Knowledge Graph Completion Models
Table 3: Comparison with other open-world KGC models on tail prediction. Note that we used the same evaluation protocol with target filtering as in ConMask. The asterisk (*) denotes that the result differs from the one published, because the MRR is calculated differently.
['Model', 'DBPedia50k H@1', 'DBPedia50k H@3', 'DBPedia50k H@10', 'DBPedia50k MRR', 'FB15k-237-OWE H@1', 'FB15k-237-OWE H@3', 'FB15k-237-OWE H@10', 'FB15k-237-OWE MRR', 'FB20k H@1', 'FB20k H@3', 'FB20k H@10', 'FB20k MRR']
Note that multiple triples with the same head and relation but different tails may occur in the dataset: (h,r,t1),... ,(h,r,tp). we remove all entities t1,...,ti−1,ti+1,... ,tp from the result list . All results (except MRR(raw)) are reported with this filtered approach. Note also that when computing the MRR, given a triple (h,r,ti) only the reciprocal rank of ti itself is evaluated (and not the best out of t1, ... ,ti,... ,tp, which would give better results). When evaluating a test triple (h,r,t), tails t′ are only included in the ranked result list if a triple of the form (?,r,t′) exists in the training data, otherwise it is skipped. We found this to improve quantitative results substantially, but it limits the predictive power of the model because tails can never be linked via new relations. For a fair comparison, all the results are evaluated using target filtering. For all models and all datasets, 200-dimensional Glove embeddings were used, except for the Complex-OWE300, which uses 300-dimensional Wikipedia2Vec embeddings. On DBPedia50k, our model performs best on all metrics except Hits@10. On FB20k it is outperformed by a small margin by ConMask but performs better on Hits@1. On FB15k-237-OWE our model outperforms all other models significantly. We believe that this is due to FB15k-237-OWE having very short descriptions. ConMask generally relies on extracting information from the description of entities with its attention mechanism, whereas our model relies more on extracting information from the textual corpus that the word embedding were trained on. This enables our model to provide good results without relying on having long descriptions.