Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models
Table 11: The detailed rating scores from AMT.
['AMT Ratings Model', 'Fluency/Consistency/Engagingness  [BOLD] Dailydialogue', 'Fluency/Consistency/Engagingness  [BOLD] Switchboard', 'Fluency/Consistency/Engagingness  [BOLD] Cornell Movie']
In addition to perplexity, we use the Amazon Mechanical Turk (AMT) platform for human evaluation of different training processes on the three dialogue data-sets. For the AMT rating, each turker is given a dialogue context, and a randomly permuted set of model sample responses. The turker is then asked to rate each sample response according to its fluency, consistency, and engagingness respectively, using an integer score from 1 to 9. The reference response is also rated for comparison. For each data-model pair, we collect 2,500 ratings. To remove bias among annotators, we use the bayesian inference code from Kulikov et al. Since we use top-k sampling, the BLEU score is not directly suitable for our setting Liu et al.