Sparse Sinkhorn Attention
Table 2: Experimental results on Language Model One Billion (LM1B) benchmark using the Base (50M parameter) and Big (430M) setting.
['Model', 'Perplexity Base', 'Perplexity Big']
On both parameter settings, Sinkhorn Transformers outperform all local attention models and Sparse Transformer. Pertaining to relative performance between Sinkhorn and local attention, the performance gain at each B setting ranges from 2−3 perplexity points. Notably, on the base setting, Sinkhorn Transformer outperforms the vanilla Transformer at B=32 and B=64. At B=16, Sinkhorn Transformers remain competitive to base Transformers. On the big setting, Sinkhorn Transformers fail to outperform vanilla Transformers, but still perform reasonably well despite being more memory efficient. Finally, the Sinkhorn Mixture model outperforms all models.