Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation
Table 2: The effect of F-F. We list the BLEU scores of Deep-Att with and without F-F. Because of the parameter exploding problem, we can not list the model performance of larger depth without F-F. For ne=1 and nd=1, F-F connections only contribute to the representation at interface part (see Eq. 3.1).
['Models', 'F-F', '[ITALIC] ne', '[ITALIC] nd', 'BLEU']
Next we list the effect of the novel F-F connections in our Deep-Att model of shallow topology in Tab. When ne=1 Since there is only a single layer, the use of F-F connections means that at the interface part we include ft into the representation (see Eq. We find F-F connections bring an improvement of 1.1 in BLEU. After we increase our model depth to ne=2 and nd=2, the improvement is enlarged to 1.4. This suggests that F-F connections provide a fast way for gradient propagation.