Semi-tied Units for Efficient Gating in LSTM and Highway Networks
Table 1: 55h LSTM system (H=500) %WERs on dev17b. A trigram LM with Viterbi (tg) or CN (cn) decoding are used.
['ID', 'System', 'tg', 'cn']
LstmSTU Experiments The experiments started by investigating different STU settings with LSTMs. All 55h LSTMs had one feedforward hidden layer placed between the LSTM layers and output layer with H=500. Two baseline systems with one LSTM layer (1L), L55h0 and L55h1, were trained, where L55h0 was a standard LSTM and L55h1 was an LSTMP with the projection size P=250. L55h4 outperformed L55h2 due to the use of distinct “peepholes”, but this also increased the training difficulty and was not used. L55h5 used untied bias vectors and was found to only improve the convergence speed by producing better criterion values in the early epochs in training and similar values in the end.