Evaluating Style Transfer for Text
Table 6: Fleiss’ kappas for human judgments of absolute naturalness and relative naturalness of texts.
['Model', 'Absolute  [ITALIC] τ=3', 'Absolute  [ITALIC] τ=2', 'Relative']
For style transfer intensity, kappas for relative scoring do not show improvement over the previously used approach of absolute scoring of x′. However, we observe the opposite for the aspect of naturalness. Despite the two types of tasks having different numbers of categories (2 vs 5), we can compare them by using a threshold τ to bin the absolute score for each text into a “natural” group (x′ is considered to be more natural than x) or “unnatural” one (vice versa), like in relative scoring. For example, τ=2 places texts with absolute scores greater than or equal to 2 into the natural group. Judgments for relative tasks yield greater inter-rater reliability than those of absolute tasks across multiple thresholds (τ∈{2,3}). This suggests that the relative scoring paradigm is preferable in human evaluations of naturalness.