Sparse Sinkhorn Attention
Table 3: Comparison with other published works on LM1B that uses per-word Perplexity. Sinkhorn Transformer remains competitive to other Transformer models and High Budget MoE models.
['Model', '# Params', 'Perplexity']
Notably, our best performing Sinkhorn Transformer remains competitive with the High Budget MoE (shazeer2017outrageously) and Evolved Transformer (so2019evolved) models. This demonstrates the overall competitiveness of Sinkhorn Transformers. Unfortunately, we were unable to outperform Mesh Tensorflow (shazeer2018mesh) on our setup, which consists of 5 billion parameters. Nevertheless, we consider our results to be reasonable given the improved memory complexity.