Training Language GANs from Scratch
Figure 4: Matching n-grams in EMNLP2017.
['[BOLD] Model', '[BOLD] World level perplexity']
A common criticism of GAN models is that they produce realistic samples by overfitting to the training set, e.g. by copying text snippets. For a selection of ScratchGAN samples we find and present the nearest neighbors present in the training set. We consider two similarity measures, a 3-gram cosine similarity — to capture copied word sequences, and a cosine similarity from embeddings produced by the Universal Sentence Encoder —to capture semantically similar sentences. Additionally we perform a quantitive analysis over the full set of samples; we also compare the longest matching n-grams between text from the training set and (a) ScratchGAN samples, (b) MLE samples, and (c) text from the validation set. We conclude the generator is producing genuinely novel sentences, although they are not always grammatically or thematically consistent.