Instructions for NAACL-HLT 2019 Proceedings
Table 4: Robustness of Representation Models, the effect of adversarial attack on link prediction task. We consider two scenario for the target triples, 1) choosing the whole test dataset as the targets (All-Test) and 2) choosing a subset of test data that models are uncertain about them (Uncertain-Test).
['[EMPTY]', '[BOLD] Models', '[BOLD] YAGO3-10  [BOLD] All-Test', '[BOLD] YAGO3-10  [BOLD] All-Test', '[BOLD] YAGO3-10  [BOLD] Uncertain-Test', '[BOLD] YAGO3-10  [BOLD] Uncertain-Test', '[BOLD] WN18  [BOLD] All-Test', '[BOLD] WN18  [BOLD] All-Test', '[BOLD] WN18  [BOLD] Uncertain-Test', '[BOLD] WN18  [BOLD] Uncertain-Test']
All-Test CRIAGE-Add outperforms the baselines, demonstrating its ability to effectively attack the KG representations. It seems DistMult is more robust against random attacks, while ConvE is more robust against designed attacks. CRIAGE-FT is more effective than CRIAGE-Add since changing the score of a fake fact is easier than of actual facts; there is no existing evidence to support fake facts. We also see that YAGO3-10 models are more robust than those for WN18. Uncertain-Test To better understand the effect of attacks, we consider a subset of test triples that 1) the model predicts correctly, 2) difference between their scores and the negative sample with the highest score is minimum. The attacks are much more effective in this scenario, causing a considerable drop in the metrics. Further, in addition to CRIAGE significantly outperforming other baselines, they indicate that ConvE’s confidence is much more robust.