Efficient Attention using a Fixed-Size Memory Representation
Table 2: BLEU scores on WMT’17 translation datasets from the memory attention models and regular attention baselines. We picked the best out of the four scoring function combinations on the validation set. Note that en-tr does not have an official test set. Best test scores on each dataset are highlighted.
['Model', 'Dataset', '[ITALIC] K', 'en-cs', 'en-de', 'en-fi', 'en-tr']
We see that our memory attention model with sufficiently high K performs on-par with, or slightly better, than the attention-based baseline model despite its simpler nature. Across the board, models with K=64 performed better than corresponding models with K=32, suggesting that using a larger number of attention vectors can capture a richer understanding of source sequences. Position encodings also seem to consistently improve model performance.