Enhancing Machine Translation with Dependency-Aware Self-Attention
Table 6: Training times (in seconds) for the Transformer baseline and Transformer+Pascal on each data set. Pascal adds negligible overhead.
['[BOLD] Corpus', '[BOLD] Transformer', '[BOLD] +Pascal']
A1SS0SSS0Px3 Training details All experiments are based on the base Transformer architecture and optimized following the learning schedule of \newcitevaswani2017attention with 8,000 warm-up steps. Similarly, we use label smoothing \epsilon_{ls}=0.1 szegedy2016rethinking during training and employ beam search with a beam size of 4 and length penalty \alpha=0.6 wu2016google at inference time. We use a batch size of 32K tokens and run experiments on a cluster of 4 machines, each having 4 Nvidia P100 GPUs.