Sparse Sinkhorn Attention
Table 5: Experimental results on pixel-wise image generation (CIFAR-10)
['Model', 'Bpd']
Our proposed Sinkhorn Transformer outperforms all baselines. The local attention model performs the worst, which can be intuitively attributed to lack of global knowledge. While keeping the local window identical, our model also outperforms Sparse Transformer which demonstrates its utility as an efficient attention method. Finally, the Sinkhorn Mixture performs worse than the ordinary Sinkhorn Transformer, suggesting that a restricted (and learned) global view may serve as a useful inductive bias.