Exploring Hyper-Parameter Optimization forNeural Machine Translation on GPU Architectures
TABLE VI: Words-per-second (average) and number of epochs, comparing activation units, learning rates and GPUs.
['cell', 'learn-rt', 'words-per-sec P100', 'words-per-sec V100', 'validation P100', 'validation V100', 'words-per-sec P100', 'words-per-sec V100', 'validation P100', 'validation V100']
When fixing learning rate, the V100 was able to execute more words-per-second than the P100, and was able to converge at an earlier iteration. When comparing hidden units, GRUs were able to execute higher words per second on a GPU and converge at a reasonable rate (at 18000 iterations) for most learning rates, except for 5e-3. When looking at LSTMs, words-per-second executed on a V100 was similar, although at a higher learning rate it was able to converge at 42000 iterations, but at the cost of longer training time and slower convergence (35000 iterations).