Generalize Symbolic Knowledge WithNeural Rule Engine
Table 3: NRE on the Criminal Case Classification dataset. With “Finetune”, NRE increases the recall of rules significantly while maintaining a high precision.
['[BOLD] Model', '[BOLD] Precision', '[BOLD] Recall', '[BOLD] F1']
Since tokens can be represented as characters (Hanzi) or words in Chinese, we conduct experiments with “NRE-Char” and “NRE-Word”, where cases and rules are based on characters and words respectively. We conduct experiments to test whether “Finetune” makes effect, and the results are listed as “NRE-Char-Finetune” and “NRE-Word-Finetune”. After integrating neural networks, the recall increases to a great extent, but the precision decreases to an unacceptable level. There is always a trade-off between precision and recall, so we adjust the decision threshold to make futher analysis of the model performances. The reasons are as follows. A RE has a hierarchical structure while the LSTM models read it in a linear way. And special characters in RE such as “.”, “?” are difficult to be modeled by a sequence model. Besides, RE is insensitive to the length of context, but a sequence model is effective in capturing local features. By integrating attention, the model is able to detect long distance dependency between words. The “LSTM-Attention-Output” outperforms “LSTM-Attention-Input” since the rule output can directly gain information from the case output, and the most useful pattern is captured by max-pooling. In brief, the LSTM baselines can not work efficiently in learning the symbolic knowledge from logic rules with hierarchical structures.