Detecting Local Insights from Global Labels: Supervised & Zero-Shot Sequence Labeling via a Convolutional Decomposition
Table 23: Token-level evaluation on the subset of the synthetic test set consisting of partially generated sentences and the corresponding human-written sentence. All model scores are from the uniCNN+BERT+S* model. †Note that the random and majority class baselines are in terms of the Original-partial set and differ slightly for the other sets due to varying sentence lengths.
['Model', 'Token-level labeling P', 'Token-level labeling R', 'Token-level labeling  [ITALIC] F1', 'Token-level labeling  [ITALIC] F0.5']
Despite decent effectiveness at detecting synthetic tokens in the static setting of the original test set (Original-partial), it is straightforward to drive detection to chance in the constrained sampling setting, requiring no more than 500 candidate samples for each sentence from which the detection model can choose. The combination of constrained sampling and slightly perturbing the sampling procedures leads to particularly severe degradation in detection, as we see in Group 2 and Group 3. In both of these sets, the model detects on average 0.05 and 0.03 synthetic tokens within the chosen candidate synthetic sentences, reflecting that the detection model can often find a sentence without a detected error among the up to 500 candidate samples. This value is 1.07 for Group 1, which explains the higher scores; however, this, too, is sufficient to make the detection F scores fall below random and majority class baselines. We assume that increasing the candidate sample pool beyond 500 would further push down the detection scores.