Automatic Detection of Generated Text isEasiest when Humans are Fooled
Table 1: Performance (accuracy and AUC) of the fine-tuned BERT classifier and several simple baselines on detecting length-192 sequences generated with one word of priming (1worccond). Note that p1.0 refers to untruncated random sampling, where we sample from 100% of the probability mass. The last column shows human performance on the same task where accuracy with a 50% baseline is computed by randomly pairing samples from each decoding strategy with a human-written sample.
['Method', 'BERT acc', 'BERT AUC', 'BagOfWords acc', 'BagOfWords AUC', 'HistGLTRBuckets acc', 'HistGLTRBuckets AUC', 'Hist50Buckets acc', 'Hist50Buckets AUC', 'TotalProb acc', 'Human acc']
Reassuringly, BERT far surpasses all simple baselines, indicating that it is not fully possible to solve the detection problem without complex sequence-based understanding. The simplest baseline, TotalProb, which makes a decision based on the likelihood of the sequence, performs surprisingly well (over 60% accuracy for all sampling methods) relative to the methods which involve training logistic regression models.