Evaluating Style Transfer for Text
Table 5: Fleiss’ kappas for human judgments of content preservation of unmasked and style-masked texts.
['Model', 'Text Modification Setting Unmasked', 'Text Modification Setting Style Masked']
The kappa metric is often levied in a relative fashion, as there are no universally accepted thresholds for agreements that are slight, fair, moderate, etc. For comprehensive experimentation, we compare kappas over the outputs of each style transfer model. The kappa score for ratings of content preservation based on style-masked texts is 0.297.