Simple Recurrent Units for Highly Parallelizable Recurrence
Table 5: Ablation analysis on SQuAD. Components are successively removed and the EM scores are averaged over 4 runs.
['Model', '4 layers', '6 layers']
We perform ablation analyses on SRU by successively disabling different components: Remove the point-wise multiplication term v⊙ct−1 in the forget and reset gates. The resulting variant involves less recurrence and has less representational capacity. Disable the scaling correction by setting the constant α=1. Remove the skip connections. We train model variants on the classification and question answering datasets.