Energy-Based Models for Text
Table 7: Cross-corpora generalization accuracy using TransfBig generator and UniT discriminator (except for the last row which used a bidirectional transformer). Each row specifies the corpora used at training time, Ctrain. Each column shows the corpus used at test time, Ctest.
['train corpora', 'test corpora Books', 'test corpora CCNews', 'test corpora Wiki']
We observe that models generalize less well across corpora; for instance, when testing on Wikitext a discriminator trained with either Books or CCNews, the accuracy is 59.1% and 65.5%, respectively. However, training on the union of two of the corpora gives a large benefit over training on just one or the other when testing on the third. To answer this question and to provide an automatic way to assess generation quality, we have tested the false positive rate, that is the fraction of machine generated samples that are deemed human generated text, using as discriminator the BiT model. We found that the baseline language model Pϕ (Base LM) has a false positive rate of 17.8% while the joint language model Pθ (Joint BiT-med) has a much higher false positive rate of 31.8%.