Temporal Attention-Gated Model for Robust Sequence Classification
Table 1: Classification accuracy (%) on Arabic spoken digit dataset by different sequence classification models. Asterisked models (∗) are trained and evaluated on the clean version of data. Note that we can customize separately the complexity of TAGM’s two modules. This design advantage is shown when looking at the optimal TAGM model (after validation) which has 128 dimensions for the Temporal Attention Module, and 64 dimensions for the Recurrent Attention-Gated Units.
['Model', '#Hidden units', '#Parameters', 'Accuracy']
While our model is similar to RNN variants like GRU and LSTM, it is specifically designed with salience detection in mind and has four key differences when compared to them: We only focus on one scalar attention score to measure the relevance of the current time step instead of generally modeling gate’s multi-dimensional values for each hidden unit as done by GRU and LSTM. We separate the attention modeling and recurrent hidden representation learning as two independent modules to decrease the degree of coupling. We employ a bi-directional RNN to take into account both the preceding and the following information of the sequence in the temporal attention module. It should be noted that it is different from the design of the gates in the bi-directional LSTM model since the latter just concatenates the hidden representations of two unidirectional LSTMs, which does not remedy the downside that all vectorial gates are still calculated by considering only one-directional information. Our model only contains one scalar gate, namely the attention gate, rather than 2 vectorial gates in GRU and 3 gates in LSTM. Doing so enforces the attention gate to take full responsibility of modeling all the salience information. In addition, the model contains fewer parameters (compared to LSTM) and simpler gate structure with less redundancy (compared to GRU and LSTM). It eases the training procedure and can alleviate the potential over-fitting and has better generalization given small amount of training data, which is demonstrated in