Efficient Attention using a Fixed-Size Memory Representation
Table 3: Decoding time, averaged across 10 runs, for the en-de validation set (2169 examples) with average sequence length of 35. Results are similar for both PE and non-PE models.
['[BOLD] Model', '[BOLD] Decoding Time (s)']
We measured decoding time over the full validation set, not including time used for model setup and data loading, averaged across 10 runs. The average sequence length for examples in this data was 35, and we expect more significant speedups for tasks with longer sequences, as suggested by our experiments on toy data. Note that in our NMT examples/experiments, K≈T, but we obtain computational savings from the fact that K≪D. We may be able to set K≪T, as in toy copying, and still get very good performance in other tasks. For instance, in summarization the source is complex but the representation of the source required to perform the task is ”simple” (i.e. all that is needed to generate the abstract).