Efficient Attention using a Fixed-Size Memory Representation
Table 1: BLEU scores and computation times with varying K and sequence length compared to baseline models with and without attention.
['[BOLD] Length', '[BOLD] Model', '[BOLD] BLEU', '[BOLD] Time (s)']
This is a study of the trade-off between computational time and representational power. A large K allows us to compute complex source representations, while a K of 1 limits the source representation to a single vector. We can see that performance consistently increases with K up to a point that depends on the data length, with longer sequences requiring more complex representations. The results with and without position encodings are almost identical on the toy data. Our technique learns to fit the data as well as the standard attention mechanism despite having less representational power. Both beat the non-attention baseline by a significant margin.