Realistic Re-evaluation of Knowledge Graph Completion Methods
Table 2. The strong FMRR↑results on a few Cartesian product relations in FB15k-237
['[BOLD] relation', '[BOLD] # of triples', '[BOLD] TransE', '[BOLD] DistMult', '[BOLD] ComplEx', '[BOLD] ConvE', '[BOLD] RotatE']
(2) Albeit not always meaningful, one may still perform link prediction on Cartesian product relations. However, a simpler approach can be more effective than learning complex embedding models. For instance, by examining all instance triples of a relation r, a method can identify the sets of all subjects S\textslr={\small h | ∃\textslr(%h,\small t)∈G} and objects O\textslr={\small t | ∃\textslr(%h,\small t)∈G} in the instance triples. By observing a large percentage of possible subject-object pairs existing in the relation, the method can derive the relation might be a Cartesian product relation. More specifically, if |\textslr| / (|S\textslr|×|O\textslr|) is greater than a pre-determined threshold (0.8 in our study), we consider r a Cartesian product relation. There are a total of 31,771 relations (375,387,927 instance triples) in the aforementioned May 2013 Freebase snapshot, of which 3,568 relations have only one instance triple each. Among the 28,203 remaining relations (375,384,359 instance triples), we detected 2,951 Cartesian product relations (2,605,338 instance triples) using this method. We also identified 142 Cartesian product relations in FB15k, with 13,038 triples. Although there are not as many Cartesian product relations as reverse relations, we discovered that among the relations on which embedding models attained the highest accuracy there are Cartesian product relations. These are the Cartesian product relations among the top-12 relations ranked by FMRR↑ of ConvE on FB15k-237.