SwitchOut: an Efficient Data Augmentation Algorithmfor Neural Machine Translation
Table 4: Hyper-parameters for our experiments.
['[BOLD] Task', '[ITALIC] nlayers', '[ITALIC] nheads', '[ITALIC] dk,  [ITALIC] dv', '[ITALIC] dmodel', '[ITALIC] dinner', 'init', 'clip', '[ITALIC] λdrop', '[ITALIC] τ−1 [ITALIC] x', '[ITALIC] τ−1 [ITALIC] y', '[ITALIC] λword']
For the WMT en-de task, we use the legacy learning rate schedule as specified by Vaswani et al. For the de-en task and the en-vi task, the learning rate is initially 0.001, and is decreased by a factor of 0.97 for every 1000 steps, starting at step 8000. All models are trained for 100,000 steps, during which one checkpoint is saved for each 2500 steps and the final evaluation is performed on the checkpoint with lowest perplexity on the dev set.