Sparse Sinkhorn Attention
Table 6: Experimental results on word and character level document classification on IMDb dataset and SST datasets.
['Model', 'IMDb Word', 'IMDb Char', 'SST Word', 'SST Char']
Sinkhorn Transformers demonstrate promising results on sentiment analysis datasets on both word and character level. Even with significant memory savings, Sinkhorn Transformers are able to outperform or remain competitive with the baseline Transformer model. We also take this chance to benchmark the Sortcut encoder, which further reduces memory complexity. On all settings, we find that the Sortcut variation can achieve similar performance to not only the standard Sinkhorn Transformer but also the vanilla Transformer.