Real or Fake? Learning to Discriminate Machine from Human Generated Text
Table 5: Generalization of BiTransf and UniTransf energy function to state of the art generators. The energy functions (discriminators) are trained on the concatenation of all three corpora (same as in table 4 ALL rows). Test time negatives are generated by models specified in the rows, with their training set in parenthesis and model size in millions of parameters. Note that both the training corpus and GPT2 generator are “unseen” by the energy function during training.
['Energy function → Test domain →', 'Energy function → Test domain →', 'UniTransf trained on ALL CCNews', 'UniTransf trained on ALL WebCorpus', 'BiTransf trained on ALL CCNews', 'BiTransf trained on ALL WebCorpus']
This is an instance of unseen generalization since Ctrain≠Ctest, and Atrain≠Atest. We also consider generations from TransfHuge (last row) whose configuration is similar to the unreleased biggest GPT2 model with 1.4 billion parameters, 7 times bigger than TransfBig, the generator used at training time. Finally, in the wild setting we explore generalization to a black-box generator. As we perform discrimination in unconditional setting, we train our energy model on all possible prefixes including an empty prefix. Evaluation is performed in the same manner as for the in-domain and cross-architecture modes.