Instructions for NAACL-HLT 2019 Proceedings
Table 6: Error Detection Accuracy in the neighborhood of 100 chosen samples. We choose the neighbor with the least value of Δ(s′,r′)(s,r,o) as the incorrect fact. This experiment assumes we know each target fact has exactly one error.
['[BOLD] Methods', '⟨ [ITALIC] s′, [ITALIC] r′, [ITALIC] o⟩ [BOLD]  Noise Hits@1', '⟨ [ITALIC] s′, [ITALIC] r′, [ITALIC] o⟩ [BOLD]  Noise Hits@2', '⟨ [ITALIC] s′, [ITALIC] r, [ITALIC] o⟩ [BOLD]  Noise Hits@1', '⟨ [ITALIC] s′, [ITALIC] r, [ITALIC] o⟩ [BOLD]  Noise Hits@2']
To evaluate this application, we inject random triples into the graph, and measure the ability of CRIAGE to detect the errors using our optimization. We consider two types of incorrect triples: 1) incorrect triples in the form of ⟨s′,r,o⟩ where s′ is chosen randomly from all of the entities, and 2) incorrect triples in the form of ⟨s′,r′,o⟩ where s′ and r′ are chosen randomly. We choose 100 random triples from the observed graph, and for each of them, add an incorrect triple (in each of the two scenarios) to its neighborhood. Then, after retraining DistMult on this noisy training data, we identify error triples through a search over the neighbors of the 100 facts. When compared with baselines that randomly choose one of the neighbors, or assume that the fact with the lowest score is incorrect, we see that CRIAGE outperforms both of these with a considerable gap, obtaining an accuracy of 42% and 55% in detecting errors.