\mathbf{\mu}-Forcing: Training Variational Recurrent Autoencoders for Text Generation
Table 1. The evaluation results on APRC test set.
['Method', 'Rec', 'KL', 'BLEU-4', 'BLEU-5', 'SLEU-4', 'SLEU-5']
We randomly sample 3,000 sentences from the test set as the references. We let each model generates 3000 sentences by sampling z from N(0,I) and calculate the average BLEU and SLEU scores (4 gram and 5 gram). Its KL loss is approximate to zero, which means this model has collapsed into a plain language model that totally ignores the latent variable z. So it has high reconstruction error. Although its BLEU score is high, its SLEU scores is 100.0, it can only generate repeated sentences which we found are almost copied from the training set. With carefully adjusting the parameter values of KL annealing and free bits, the model VRAE + KLA, VRAE + FB, and VRAE + FB-all have achieved non-zero but still small KL cost. The reconstruction error of them are also very high, which indicates that these models still ignore the latent variable z in most cases. Moreover, the SLEU scores of them are also much higher than other methods.