Evaluating Style Transfer for Text
Table 10: Percent agreement between adversarial classifiers and human scores on the naturalness of texts.
['Model', 'Unigram Adv. Clf. Absolute', 'Unigram Adv. Clf. Relative', 'Neural Adv. Clf. Absolute', 'Neural Adv. Clf. Relative']
We also confirm that sentence-level perplexity is not an appropriate metric. It exhibits no significant correlation with human scores (α=0.05). These results suggest that adversarial classifiers can be useful for automating measurement of naturalness.