Data Distillation for Controlling Specificity in Dialogue Generation
Table 6: Adversarial success and machine-vs-random accuracy for Iter1 which always generating response using the model trained on the full set, random which randomly samples a model for generation, and the proposed model.
['model', '[ITALIC] AdverSuc', '[ITALIC] machine-vs-random']
Adversarial success (AdverSuc) refers to the percentage of machine-generated responses that are able to fool an trained evaluator model into believe that they are generated by humans; machine-vs-random accuracy denotes the accuracy of a trained evaluator model (a different evaluator from the one used in adversarial success) at distinguishing between machine-generated responses and human utterances randomly sampled without regard for the input. Superior models should obtain higher values of both adversarial success and machine-vs-random accuracy. We refer readers to \newciteadd for more details. We observe that the RL model performs better than always using the model trained on the full dataset (Iter1) or choosing a distillation model at random (as one would expect, since the RL model is trained to optimize adversarial success).