Improving Conditional Sequence Generative Adversarial Networks by Stepwise Evaluation
TABLE II: Results of synthetic experiment - counting.
['[EMPTY]', '[BOLD] Prec', '[BOLD] SampP', '[BOLD] SampR', '[BOLD] FKLD', '[BOLD] IKLD', '[BOLD] FKLD+IKLD']
In the synthetic experiments, because we know all the possible answers to an input sentence, we can compute the precision and recall rates. Given each input x, we used the generator G to generate one sample by argmax and 100 samples by sampling from the probability distribution of softmax outputs. It is not possible to evaluate recall for argmax because there is only one sample. (2) SampP, the precision of softmax outputs. (3) SampR, the recall of softmax outputs. They are intuitive indexes to test the performance of the models. We compared StepGAN with other stepwise evaluation methods including REGS, MCTS, MaskGAN. In addition, we performed weighted stepGANs (stepGAN-W) that we gave decaying weights for the policy gradient. We had tested both increasing and decaying cases, and found that increasing weights did not improve the training, so the results of increasing weights were not shown here. Precision and Recall. Specifically, SeqGAN improves the model by 0.94; previous stepwise evaluation methods improve the model by 1.19∼3.68; our proposed approaches StepGAN and StepGAN-W improve the model by 3.87 and 5.97 respectively. Especially, the MaskGAN increases the precision the most but significantly drops the recall. This indicates that MaskGAN overfits to a smaller portion of possible answers. The results show that StepGAN-W not only achieves better performance than other approaches, it is also more stable with different parameter initialization.