Improve Diverse Text Generation by Self Labeling Conditional Variational Auto Encoder
Table 2: Results on EGOODS. The bottom 3 lines are CVAE based methods.
['Methods', 'BLEU-prec', 'BLEU-recall', '[ITALIC] distinct-1', '[ITALIC] distinct-2']
, experiments are conducted on our newly collected native one-to-many dataset EGOODS. Performances of different methods are shown in Table. First of all, our method achieves comparable accuracy with baselines and best diversity among all methods with an only exception of SEQ2SEQ+noise(0.8) on distinct-2. This demonstrates the effectiveness of SLCVAE on the one-to-many data. Although SEQ2SEQ+noise(0.8) gets the best distinct-2, its precision is sacrificed significantly due to the nosie, which means the results tend to be meaningless. In detail, our method harvests the much better gains on word-level diversity while is only slightly better than CVAE on BLEU-recall. We explain this in two folds: First, strong baselines can benefit from the large scale and one-to-many nature of EGOODS to better fit the multiple targets. Another reason is that automatically evaluating the quality of generated texts is very challenging. BLEU-recall only measures the coverage of hypothesis for the annotated targets, and could not judge good algorithms precisely when the annotations are limited. In such situation, distinct measures the vocabulary a model actually uses and demonstrates its absolute lexical diversity. Example results will show this in next subsection.