The Daunting Task of Real-World Textual Style Transfer Auto-Evaluation
Table 1: Results on Yelp “style” (sentiment) transfer. BLEU is between 1000 transferred sentences and human references, and accuracy is restricted to the same 1000 sentences. Accuracy: post-transfer style classification accuracy (by a classifier pretrained on the two corpora). CAE†: cross-aligned autoencoder as in shen-1. BLEU scores reported for simple-transfer are copied from evaluations by yang2018unsupervised. Note that if a model name appears twice, the models are from different stopping points during training.
['Model', 'BLEU', 'Accuracy']
First, we show that one intuitive way of evaluating style transfer is inadequate: computing BLEU scores (papineni2002bleu) between generated/transferred outputs and human-written gold-standard outputs. In fact, simple-transfer crowdsourced 1000 Yelp human-written references as test data (500 positive-sentiment sentences transferred from negative sentiment, and 500 negative-sentiment sentences transferred from positive sentiment).