Robust Training of Vector QuantizedBottleneck Models
TABLE II: On ScribbleLens, data-dependent codebook reestimation results in the fullest codebook usage, leading to the best reconstruction measured in bits per dim. Uniform and unigram ELBOs show that increased bottleneck bandwidth results in a smaller reconstruction likelihood improvement.
['Model', 'Rec. BPD', 'Pplx', 'Uniform NELBO bits', 'Unigram NELBO bits']
Used alone, batch normalization has a minor detrimental effect. Again, improvements are related to a higher usage of the bottleneck capacity. logp(x)=log∑zp(x|z)p(z)≥ELBO==logEz q(z|x)p(x|z)−KL(q(z|x)||p(z))==log(p(x|z=q(x))+logp(z =q(x)). (9) The last transition follows from the fact that q(x) gives a deterministic encoding, p(x|z) is the reconstruction loss, and p(z) is the prior latent code probability. We consider two formulations of the prior over latent vectors: a uniform prior p(z)=1/K, and a unigram prior of observed codeword frequencies. The negative of ELBO (NELBO) has an intuitive interpretation: to transmit a data sample, we need to transmit the information contained in the latent features plus the information required to reconstruct the data sample from the conditional decoder probability. The Uniform NELBO is computed as BPD+log2(4096)/32/4, i.e. the cost of reconstructing a single pixel augmented by the cost of transmitting a single latent amortized over all pixels this latent corresponds to, while for Unigram NELBO we account for the actual codebook usage by computing BPD+log2(perplexity)/32/4.