Detecting Local Insights from Global Labels: Supervised & Zero-Shot Sequence Labeling via a Convolutional Decomposition
Table 22: Token-level evaluation on the full synthetic test set, consisting of a mix of original sentences and fully- and partially- generated sentences, where each token is labeled as human written or machine generated.
['Model', 'Token-level labeling P', 'Token-level labeling R', 'Token-level labeling  [ITALIC] F1', 'Token-level labeling  [ITALIC] F0.5']
In the interest of brevity, we focus on the supervised model for token-level evaluation, since, by definition of this task, we have access to the token-level labels (since we generated the sentences) for training (the models, of course, do not have access to the true labels for inference), where each token is labeled as human written (class 0) or machine generated (class 1). * model is reasonably effective at detecting synthetic text at the token-level. To get a qualitative sense of the nature of the full training, dev, and test sets, with fully- and partially- generated sentences, we include examples of the labeled output in Appendix C. Qualitatively, out of context, it is often not obvious that a particular sentence is synthetic when the uniCNN+BERT+S* correctly marks a token as synthetic, reflecting our intuition that the model may be able to pick up on relatively subtle distributional differences in diction, syntax, and length seen in the training set.