Assessing the Stylistic Properties of Neurally Generated Text in Authorship Attribution
Table 1: Mean F1, Precision (P) and Recall (R) scores for all classification experiments.
['Source', 'Experiment', 'F1', 'P', 'R']
We focus on the macro-averaged F1-scores in our discussion, although one should not forget that the scores vary considerably over individual authors (cf. Fig. With respect to the authentic data, classifying α on the basis of ω is slightly more difficult than the reverse direction, which seems a negligible directionality artifact. When we use the generated data as training material to classify authentic material  , we see that the F1-scores drop significantly for both LMs, although the NGLM seems more robust in this respect. Interestingly, the drop is much less significant for the opposite situation, where we train on authentic material and classify generated material . This suggests that enough stylistic information is preserved in the generated text to attribute it to the original author, but that this information in isolation does not suffice to train a convincing attribution system on. When used in isolation, the NGLM outperforms the RNNLM in both setups. However, the situation is clearly different for the augmentation or self-learning setup ()---c.f. Here, we see that the RNNLM performs better than the NGLM in the corresponding experiment -- the NGLM in fact even performs worse in this case than in the normal  setup.