SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection
Table 1: BLEU scores on the newstest2013 for development and newstest2014 for test for WMT English-German. L denotes the length of the input sequence.
['[BOLD] Model', '[BOLD] H (heads)', '[BOLD] N (blocks)', '[BOLD] edges', '[BOLD] dev (BLEU)', '[BOLD] test (BLEU)', '[BOLD] test (cased sacreBLEU)']
As we gradually increase the number of edges for each layer (from 2 to 5 to 10 to 20 per node), we can see that the performance first increases, reaching the highest with α set to 10, and then decreases. This means that performing attention operations between all pairs is not only unnecessary, but can hurt the performance. Memory saved from sparse connections allow for more heads to perform attentions and deeper networks with more blocks, leading to better performances over vanilla transformers. Relative positions between nodes in the dependency trees are encoded in distance encodings of the edge predictor. The introduction of dependency parser for attention construction introduces +0.14 BLEU score boost. We did not observe significant performance boost from the head-adaptive strategy, and thus omit their performances.