Sparse Sinkhorn Attention
Table 4: Experimental results on character level language modeling on LM1B with sequence lengths of 1024 characters.
['Model', 'Bytes per char (Bpc) Base', 'Bytes per char (Bpc) Big']
On both settings (base/big), our proposed Sinkhorn Transformer outperforms both local attention and Sparse Transformer, which affirms its effectiveness as an efficient attention method. On the contrary, local attention performs substantially worse compared to its counterparts, likely due to not having much global context. From this set of experiments, the vanilla full attention Transformer outperforms all efficient attention methods. However, our Sinkhorn Mixture model outperforms the Transformer baseline, achieving the best performance for both parameterizations.