Ruminating Reader: Reasoning with Gated Multi-Hop Attention
Table 3: Layer ablation results. The order of the listing corresponds to the description in Appendix A.1. CRL refers to context ruminate layer and QRL refers to query ruminate layer.
['[BOLD] Ablation Variant', '[BOLD] Dev F1', '[BOLD] Dev EM']
The ablation experiments show how each component contribute to the model. Experiments 3 and 4 show that the two ruminate layers are both important and helpful in contributing performance. It is worth noting that the BiLSTM in the context ruminate layer contributes substantially to model performance. We find this somewhat surprising, since it takes the same input in each timestep, but it nonetheless successfully digests the summarization information representation and produces a useful input for the gating component. Experiments 7 and 8 show that the modeled summarization vector representation can provide information to gates reasonably well. The drop in performance in both experiments 9 and 10 shows that the key information for new query and context representation are the are first stage query and context encodings. Experiments 11 and 12 shows that the summarization vector representation does help the later stage of reasoning.