Effective Estimation of Deep Generative Language Models
Table 4: Performance on the Yahoo/Yelp corpora. Top rows taken from (He et al., 2019)
['Model', 'Yahoo  [ITALIC] R', 'Yahoo NLL↓', 'Yahoo PPL↓', 'Yahoo AU↑', 'Yelp  [ITALIC] R', 'Yelp NLL↓', 'Yelp PPL↓', 'Yelp AU↑']
, we report results on the Yahoo and Yelp corpora Yang et al. We compare to the work of He et al. We report on models trained with the standard prior as well as an MoG prior both optimised with MDR, and a model trained without optimisation techniques. It can be seen that MDR compares favourably to other optimisation techniques reported in He et al. Whilst aggressive training of the inference network performs slightly better in terms of NLL and leads to more active units, it slows down training by a factor of 4. The MoG prior improves results on Yahoo but not on Yelp. This confirms that targeting a pre-specified rate leads to VAEs that are not collapsed without hurting NLL.