Decompressing Knowledge Graph Representations for Link Prediction
Table 3: Performance of different models w/ and w/o decompressing on the testset of WN18RR dataset.
['[BOLD] #', 'Model', '[ITALIC] de', '^ [ITALIC] de', '[ITALIC] dr', '^ [ITALIC] dr', '[BOLD] WN18RR MRR', '[BOLD] WN18RR H@1', '[BOLD] WN18RR H@3', '[BOLD] WN18RR H@10']
People may be encouraged to use larger embedding size by above observations, but it is not just modeling scalability that sets them back. The same phenomenon has been observed for word embeddings, and \citeauthoryin2018dimensionality \shortciteyin2018dimensionality explains this phenomenon under the bias-variance tradeoff framework: larger embedding size leads to decreased bias (better reconstruct the factorized coorcurrence matrix), but increased variance (overfit to the noise in the matrix). The same analysis can be applied to knowledge graph embedding as well, considering RESCAL, for example, what it essentially does is a tensor decomposition X =ERET where X∈R|E|×|R|×|E| is the tensor that represents the training graph, Xijk=+1 if there is a relation j from entity i to entity k, and Xijk=−1 if there is none. E∈R|E|×d is the entity embedding matrix where Ei⋅ is the embedding vector of the ith entity and d is the embedding size. R∈Rd×|R|×d is the relation tensor where R⋅j⋅ is the embedding matrix of the j relation. This decomposition can be lossless with large enough embedding size, but if we do obtain such an embedding, the performance on evaluation and test sets will be zero - as the training graph tensor X is corrupted from the true graph tensor ^X by randomly flipping some of its entries. Moreover, for the link prediction task, we are exclusively evaluating those corrupted entries. It is popular for recent studies to prove that their model is fully expressive (unbiased with large enough d), but it may not be relevant to actual model performance, as the variance plays an important role here. Since DistMult is a special case of RESCAL and ComplEx generalizes DistMult to complex space, this analysis can be applied to them, and other multiplicative models, as well.