Polite Dialogue Generation Without Parallel Data
Table 4: MTurk human evaluation results on politeness level of 3 LFT models, for both the continuous and the discrete versions.
['[EMPTY]', 'Continuous-LFT', 'Discrete-LFT']
We also present results on a 3-way politeness level comparison MTurk study among the Polite-LFT, Neutral-LFT, and Rude-LFT models, i.e., the LFT model with three levels (scores) of scaling the prepended style label, corresponding to politeness scores 1.0, 0.5 and 0.0, respectively (Table. The table shows that Polite-LFT is significantly more polite than Neutral-LFT (stat. significance of p<0.01), and Neutral-LFT is in turn more polite than Rude-LFT (stat. significance of p<0.01). For inter-annotator agreement on this 3-way LFT study, we get a Kappa of 0.51 (moderate), and 0.61 (substantial) for the collapsed-Likert case. We also experimented earlier with a discrete version of LFT, where we treated responses in the [0.8,1.0] range as polite, [0.2,0.8] as neutral, and [0.0,0.2] as rude. The human evaluation scores for that model were 3.52, 3.09 and 2.93, respectively, which features less score difference between neutral and rude (Table. -S6T4 Discrete-LFT column).