Shortcut Sequence Tagging
Table 2: Comparsion of shortcut topologies. We use ~hlt to represent the original cell output of LSTM block, which equals o⊙tanh(clt), similar to ~clt:=i⊙slt+f⊙ct−1.
['[BOLD] Case', '[BOLD] Variant', '[BOLD] Dev', '[BOLD] Test']
Here we design two kinds of models for comparison: one is both clt and hlt connected through shortcuts (Case 1), the other is using m to replace clt (Case 2), as defined in the shortcut block. We find the skip connections to both the internal states and the cell outputs with multiplicative gating achieves the highest accuracy (case 1, 95.08%) on the test set. But case 2 can get a better validation accuracy (94.77%). We prefer to use case 2 since it generalizes well and much easier to train.