A Heterogeneous Graph with Factual, Temporal and Logical Knowledge for Question Answering Over Dynamic Contexts
Table 2: Results on document-level task. Our approach is abbreviated as ProGraph. ProComp is the symbolic-based baseline and other systems are strong neural baselines.
['Model', 'Performance P', 'Performance R', 'Performance F1']
We also report the performance of our re-implemented ProComp system, with some modifications as described previously. We compare ProGraph with the pure symbolic-based system ProComp described before. As shown in the table, our system achieves 17.70% absolute improvements in Recall and 13% improvement on the F1 compared with ProComp. Moreover, our system also outperforms previous strong neural baselines with 61.00% F1 score. Our model also outperforms KG-MRC, the most related work to ProGraph, with 4.23% absolute performance gain in F1 score. This observation indicates that integrating graph neural network with the symbolic system not only persists the in-domain reasoning ability of the symbolic system but also alleviates its shortcomings when learning the fluidity of concepts. To make further analysis about the effect of different components (i.e. variants of attributes, nodes or relations), we make ablation experiments. We eliminate the graph-based representation learning model. Instead of predicting the state of the participant by its learned representation from the graph network, we replace it with a paragraph-based representation calculated by max-pooling over the contextual representation ˜HDt. As shown in the table, eliminating the component of graph-based representation learning model from the overall model causes substantial performance drops (61.0% to 53.4% on F1). The result verifies that incorporating the graph network can enhance the system’s ability to perform entity state reasoning.