A Probabilistic Formulation of Unsupervised Text Style Transfer
Table 4: Comparison of gradient approximation on the sentiment transfer task.
['[BOLD] Method', '[BOLD] train ELBO↑', '[BOLD] test ELBO↑', '[BOLD] Acc.', '[BOLD] BLEU [ITALIC] r', '[BOLD] BLEU [ITALIC] s', '[BOLD] PPLD1', '[BOLD] PPLD2']
Greedy vs. Sample-based Gradient Approximation. In our experiments, we use greedy decoding from the inference network to approximate the expectation required by ELBO, which is a biased estimator. The main purpose of this approach is to reduce the variance of the gradient estimator during training, especially in the early stages when the variance of sample-based approaches is quite high. As an ablation experiment on the sentiment transfer task we compare greedy and sample-based gradient approximations in terms of both train and test ELBO, as well as task performance corresponding to best test ELBO. After the model is fully trained, we find that the sample-based approximation has low variance. With a single sample, the standard deviation of the EBLO is less than 0.3 across 10 different test repetitions. All final reported ELBO values are all computed with this approach, regardless of whether the greedy approximation was used during training. The reported ELBO values are the evidence lower bound per word.