Robust Training of Vector QuantizedBottleneck Models
TABLE III: Unsupervised bits/dim (BPD) on CIFAR-10 test set for 8×128-codeword and 5×1024-codeword VQ bottlenecks (median of five runs)
['[BOLD] Model', 'Reconstruction BPD 8×128', 'Reconstruction BPD 5×1024']
Each encoded e(x) is split into equally-sized blocks that are quantized independently. This allows us to achieve high bottleneck bitrates with small codebooks, facilitating k-means training, which increases in difficulty with the number of centroids. For this model, placing a BN layer before the quantization leads to minor improvements in BPD in all settings. All models ultimately converge to similar values. The initial high values of BPD seen in the figure is caused by Polyak checkpoint averaging.