Instructions for NAACL-HLT 2019 Proceedings
Table 3: Ranking modifications by their impact on the target. We compare the true ranking of candidate triples with a number of approximations using ranking correlation coefficients. We compare our method with influence function (IF) with and without Hessian, and ranking the candidates based on their score, on two KGs (d=10, averaged over 10 random targets). For the sake of brevity, we represent the Spearman’s ρ and Kendall’s τ rank correlation coefficients simply as ρ and τ.
['[BOLD] Methods', '[BOLD] Nations  [BOLD] Adding', '[BOLD] Nations  [BOLD] Adding', '[BOLD] Nations  [BOLD] Removing', '[BOLD] Nations  [BOLD] Removing', '[BOLD] Kinship  [BOLD] Adding', '[BOLD] Kinship  [BOLD] Adding', '[BOLD] Kinship  [BOLD] Removing', '[BOLD] Kinship  [BOLD] Removing']
CRIAGE performs comparably to the influence function, confirming that our approximation is accurate. Influence function is slightly more accurate because they use the complete Hessian matrix over all the parameters, while we only approximate the change by calculating the Hessian over eo. The effect of this difference on scalability is dramatic, constraining IF to very small graphs and small embedding dimensionality (d≤10) before we run out of memory. As it shows, CRIAGE is mostly unaffected by the number of entities while IF increases quadratically. Considering that real-world KGs have tens of thousands of times more entities, making IF unfeasible for them.