Learning to Encode Position for Transformer with Continuous Dynamical Model
Table 6: Performance comparison on WMT14 En-De data and Transformer-base architecture. Both BLUE scores and the number of trainable parameters inside each position encoder are included.
['[EMPTY]', 'BLEU\xa0(↑)', '#Parameters\xa0(↓)']
Recurrent neural network (RNN) is commonly used to perform sequential modeling. Computing the value at the i-th step relies on the results at the (i−1)-st step. Further, they all contain trainable parameters, allowing them to adapt to each particular task. Lastly, they can be extrapolated to any length as needed. To see if RNN works equally well, we model the sequence {pi}i∈{1,2,…} with RNN models: pi+1=RNN(zi,pi), (14) where zi∈Rdin is the input to the RNN model at index i. Recall in RNN language models , zi is the word embedding or hidden feature of the i-th token. In our case, since we apply RNN to learn the encodings as opposed to hidden features, sensible inputs can be scalar value i or vectorized value Vectorize(i) by sinusoidal encoding. We tried both choices on WMT14 En-De data and found that vectorized value generally works better, though not as good as our FLOATER model.