Self-attention based end-to-end Hindi-English Neural Machine Translation
Table 2: Number of Trainable parameters in each model
['[BOLD] Model', '[BOLD] Trainable parameters']
For the attention based RNN search, we are using GRU cell for computation since attention mechanism is already employed and will capture long term dependencies explicitly using attention value. GRU cell is computationally efficient in terms of computation as compared with LSTM cell. Like in our sequence-to-sequence model, the embedding layer, hidden layer is taken to be of 512 dimension, both encoder and the decoder part contains 2 as the number of hidden layers. For regularization, we are using dropout and set the value to be 20 percent. Batch size of 128 is taken. For the self attention Transformer network, we are using hidden and embedding layer to be of size 512, for each encoder and decoder, we fix the number of layers of self-attention to be 4. In each layer, we assign 8 parallel attention heads and the hidden size of feed forward neural network is taken to be 1024 in each cell. attention dropout and residual dropout is taken to be 10 percent.