Semi-tied Units for Efficient Gating in LSTM and Highway Networks
Table 3: 275h system (H=1000) %WERs on dev17b. A tri-gram LM with Viterbi (tg) or CN (cn) decoding are used.
['ID', 'System', 'tg', 'cn']
In order to ensure that the 55h results and findings can scale to a significantly larger training set, some selected LSTM and highway networks were built on the full 275h set. The hidden layer size H and LSTMP projection size P were increased to 1000 and 500, which quadrupled the parameters to better model the full training set. Comparing to S275h2 and R275h2, the sigmoid and ReLU highway networks, S275h4 and R275h4, had a 4.0% and a 3.7% WERRs respectively by increasing the model depths from 7L to 15L. All of the LSTMSTU and sigmoid/ ReLU HighwaySTU systems produced similar WERs to their corresponding conventional LSTMP and highway networks while using fewer than 40% of the parameters in the hidden layers. This validates our previous finding on a larger data set that the proposed STU can work as well as the widely used traditional gating units with far fewer parameters. The STU approach is a highly efficient way to perform general gating and information merging that can also be applied to other gated models, such as GRUs, recurrent highway networks, quasi-RNNs, and highway LSTMs etc.