Enhancing Machine Translation with Dependency-Aware Self-Attention
Table 7: Hyperparameters for the reported models. lr denotes the maximum learning rate, (\beta_{1},\beta_{2}) are Adam’s decay rates, h_{P} is the number of Pascal heads, and q is the parent ignoring probability.
['[BOLD] Corpus', 'lr', '(\\beta_{1},\\beta_{2})', 'h_{P}', 'q']
\newciteraganato-tiedemann-2018-analysis found that only a few attention heads per layer encoded a significant amount of syntactic dependencies. Our study shows that the Transformer model can be improved by having more attention heads learn syntactic dependencies.