Hierarchical Quantized Representations for Script Generation
Table 1: Negative log-likelihood (NLL) and perplexity (PPL) measures on the validation and test set. Lower is better for both metrics.
['System', 'validation NLL', 'validation PPL', 'test NLL', 'test PPL']
As our proposed models are essentially language models, it is natural to evaluate their perplexity scores, which can be viewed as an indirect measure of how well the models can identify scripts. We compute per-word perplexity and per-word negative log likelihood on the validation and test sets. We compute these values without considering the end-of-sentence (EOS) token. A good language model should assign low perplexity (high probability) to the validation and test sets. We observe that HAQAE achieves the minimum negative log likelihood and perplexity scores on both the validation and test sets as compared to the previous RNN-based models. The result is particularly interesting as autoencoders usually perform worse or comparable to other RNN language models in terms of perplexity (negative log likelihood) as is in the case of the vanilla VAE here; similar observations have also been made in Bowman et al.