SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection
Table 4: Results of image classification on CIFAR-100 using the Wide-ResNet 28-10 Zagoruyko and Komodakis (2016) as the backbone and on ImageNet using the ResNet-50 He et al. (2016) model.
['[EMPTY]', '[BOLD] CIFAR-100 GFlops', '[BOLD] CIFAR-100 top-1', '[BOLD] CIFAR-100 top-5', '[BOLD] CIFAR-100 Params', '[EMPTY]', '[BOLD] ImageNet GFlops', '[BOLD] ImageNet top-1', '[BOLD] ImageNet top-5', '[BOLD] ImageNet Params']
Specifically, the top-1 score increases from 81.6 to 82.4 for CIFAR-100 and from 77.7 to 78.7 for ImageNet. The improvement validates the importance of performing necessary attention operations under memory limit.