Training Language GANs from Scratch
Table 4: FED sensitivity on EMNLP2017 News.
['[BOLD] Model', '[BOLD] FED']
Despite the high variance of REINFORCE gradients and the often unstable GAN training dynamics, our training procedure is very stable, due to the use of large batch sizes and chosen reward structure. The low variance across hyperparameters shows that ScratchGAN is not sensitive to changes in learning rate, REINFORCE discount factor, regularization or LSTM feature sizes, as long as these are kept in a reasonable range. When we fixed hyperparameters and repeated an experiment across 50 seeds, we obtained very similar FED score; no divergence or mode collapse occurred in any of the 50 runs. For WikiText-103, the results are similar (0.055 ± 0.003). Here we clarify the definition of the subset of hyper-parameter space that we used to show the stability of our training procedure.