A Good Sample is Hard to Find: Noise Injection Sampling and Self-Training for Neural Language Generation Models
Table 6: Human correctness and quality judgments (%). Comparisons are better than (>), equal to (=), and worse than (
['Model', 'Correct. >', 'Correct. =', 'Correct. ', 'Quality >', 'Quality =', 'Quality ']
Laptops Dataset We had the same annotators look at 100 random Inform DAs from the Laptops test set since they are the majority DA type and we could use the same annotator guidelines from the E2E experiment. We do not have access to the Slug or SCLSTM outputs on this dataset, so we compared to one of the two test set reference sentences (picking at random) vs. the p1/qR with greedy decoding. Despite the low BLEU scores, we find our outputs to be of comparable quality to references 91% of the time. Moreover, they are equally as correct as the human references 100% of the time. Annotators agreed 99% and 87% of the time on correctness and quality respectively.