Improving Multi-Head Attention with Capsule Networks
Table 1: Case-insensitive BLEU scores for Zh→En translation. “# Para” denotes the number of parameters. “*” is used to indicate the improvement is statistically significant with ρ<0.05 [5].
['[BOLD] Zh→En', '# Para.', 'MT03', 'MT04', 'MT05', 'MT06', 'MT08', 'AVE.']
Both of our models (Row 4,5) with the proposed capsule network attention mechanism can not only outperform the vanilla Transformer(Row 3) but also achieve a competitive performance compared to the state-of-the-art systems(Row 1,2, we use the results from the related paper directly), indicating the necessity and effectiveness of the proposed method. It shows that our method can get the information well combed and preserve all the deserved information.