NLProlog: Reasoning with Weak Unification for Question Answering in Natural Language
Table 1: Accuracy scores in percent for different predicates on the development set of the respective predicates. +/- denote independent modifications to the base algorithm.
['[BOLD] Model', '[BOLD] MedHop', '[ITALIC] publisher', '[ITALIC] developer', '[ITALIC] country', '[ITALIC] recordlabel']
For all predicates but developer, NLProlog strongly outperforms all tested neural QA models, while achieving the same accuracy as the best performing QA model on developer. We evaluated NLProlog on the hidden test set of MedHop and obtained an accuracy of 29.3%, which is 6.1 pp better than FastQA and 18.5 pp worse than BiDAF. As the test set is hidden, we cannot diagnose the exact reason for the inconsistency with the results on the development set, but observe that FastQA suffers from a similar drop in performance. To study the impact of the rule-based reasoning on the predictive performance, we perform an ablation experiment in which we train NLProlog without any rule templates. On three of the five evaluated data sets, performance decreases markedly when no rules can be used and does not change on the remaining two data sets. This indicates that reasoning with logic rules is beneficial in some cases and does not hurt performance in the remaining ones. In a qualitative analysis, we observed that in many cases multi-hop reasoning was performed via aligning entities and not by applying a multi-hop rule. To gauge the extent of this effect, we evaluate an ablation in which we remove the MLP on top of the entity embeddings. Interestingly, the observed performance degradation is much worse than when training without rules, suggesting that much of the reasoning is actually performed by finding a suitable transformation of the entity embeddings.