Reinforced Mnemonic Reader for Machine Reading Comprehension
Table 4: Ablation study on SQuAD dev set.
['[BOLD] Configuration', '[BOLD] EM', '[BOLD] F1', 'Δ [BOLD] EM', 'Δ [BOLD] F1']
Firstly, ablation (1-4) explores the utility of reattention mechanism and DCRL training method. We notice that reattention has more influences on EM score while DCRL contributes more to F1 metric, and removing both of them results in huge drops on both metrics. Replacing DCRL with SCST also causes a marginal decline of performance on both metrics. Next, we relace the default attention function with the dot product: f(u,v)=u⋅v (5), and both metrics suffer from degradations. (6-7) shows the effectiveness of heuristics used in the fusion function. Removing any of the two heuristics leads to some performance declines, and heuristic subtraction is more effective than multiplication. Ablation (8-9) further explores different forms of fusion, where gate refers to o=g∘~x and MLP denotes o=~x in Eq. In both cases the highway-like function has outperformed its simpler variants. Finally, we study the effect of different numbers of aligning blocks in (10-12). We notice that using 2 blocks causes a slight performance drop, while increasing to 4 blocks barely affects the SoTA result. Interestingly, a very deep alignment with 5 blocks results in a significant performance decline.