Effective Estimation of Deep Generative Language Models
Table 3: Performance on the PTB test set for different priors (N, MoG, Vamp). Standard deviations of D, R, and Accgap are at most 0.1.
['Model', '[ITALIC] D', '[ITALIC] R', 'PPL↓', 'AU↑', 'Accgap']
Second, we compare the impact of expressive priors. All models are trained with a target rate of 5 using MDR, with settings otherwise the same as the previous experiment. One may wonder whether stronger priors allow us to target higher rates without hurting PPL. This does not seem to be the case: as we increase rate to 50, all models perform roughly the same, and beyond 20 performance degrades quickly. The models did, however, show a further increase in active units (VampPrior) and accuracy gap (both priors).