Widening the Representation Bottleneck in Neural Machine Translation with Lexical Shortcuts
Table 5: Model size and training speed of the compared transformer variants.
['Model', '# Parameters', 'Words / sec.']
All models are trained concurrently on four Nvidia P100 Tesla GPUs using synchronous data parallelization. Delayed optimization Saunders et al. Each transformer-BASE model is trained for a total of 150,000 updates, while our transformer-BIG experiments are stopped after 300,000 updates. Validation is performed every 4000 steps, as is check-pointing. Training base models takes ∼43 hours, while the addition of shortcut connections increases training time up to ∼46 hours (∼50 hours with feature-fusion). Parameters are given in thousands, while speed is averaged over the entire training duration.