Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection
Table 5: Results of the ablation study of the sparsification at different phases on the En-Vi test set. “Base” denotes vanilla Transformer. “T” denotes only adding the sparsification in the training phase, and “T&P” denotes adding it at both phases as the implementation of Explicit Sparse Transformer does.
['[BOLD] Task', '[BOLD] Base', '[BOLD] T', '[BOLD] T&P']
We are surprised to find that only adding the sparsification in the training phase can also bring an improvement in the performance. The improvement of 0.3 BLEU scores shows that vanilla Transformer may be overparameterized and the sparsification encourages the simplification of the model.