A Batch Normalized Inference Network Keeps the KL Vanishing Away
Table 8: Results on Yahoo and Yelp datasets. We report mean values across 5 different random runs. ∗ indicates the results are from our experiments, while others are from previous report.
['[BOLD] Model', 'Yahoo  [BOLD] NLL', 'Yahoo  [BOLD] KL', 'Yahoo  [BOLD] MI', 'Yahoo  [BOLD] AU', 'Yelp  [BOLD] NLL', 'Yelp  [BOLD] KL', 'Yelp  [BOLD] MI', 'Yelp  [BOLD] AU']
We investigate the training procedure for different models. We plot the MI Iq, DKL in the ELBO and the distance between the approximated posterior and the prior, DKL(qϕ(z)||p(z)). As in Eq. 4 in the main paper, DKL in the ELBO is the sum of the other two. Fig. Although DKL is the upper bound of the mutual information, we notice that the gap is usually large. In the initial training stage, DKL increases in the basic VAE with annealing, while its MI remains small. With the weight decreases, the method finally suffers from posterior collapse. In contrast, our approach can obtain a high MI with a small DKL value like aggressive VAE.