Bidirectional Attentive Memory Networks for Question Answering over Knowledge Bases
Table 2: Ablation results on the WebQuestions test set. Gold topic entity is assumed to be known.
['Methods', 'Macro  [ITALIC] F1']
We now discuss the performance impact of the different modules and strategies in BAMnet. Note that gold topic entity is assumed to be known when we do this ablation study, because the error introduced by topic entity prediction might reduce the real performance impact of a module or strategy. As we can see, when turning off the two-layered bidirectional attention network, the model performance drops from 0.557 to 0.534. Among all submodules in the attention network, the importance module is the most significant since the F1 score drops to 0.540 without it, thereby confirming the effectiveness of modeling the query-to-KB attention flow. On the flip side, the importance of modeling the KB-to-query attention flow is confirmed by the fact that replacing the KB-aware attention module with self-attention significantly degrades the performance. Besides, the secondary attention layer, the enhancing module, also contributes to the overall model performance. Finally, we find that the topic entity delexicalization strategy has a big influence on the model performance while the constraint delexicalization strategy only marginally boosts the performance.