Sparse Sinkhorn Attention
Table 1: Evaluation on Algorithmic Sequences of length 256.
['Model', 'Edit Dist.', 'EM']
Sinkhorn Transformers outperform all other Transformer variants, including the vanilla attention model. Sparse Transformer outperforms dense attention, which demonstrates the usefulness of sparse inductive biases. The local attention performs the worst, which demonstrates that some extent of global knowledge is required to solve this task.