SentenceMIM: A Latent Variable Language Model
Table 6: Entropy of the latent/hidden distribution for sMIM and AE (estimated using NN entropy estimator (PhysRevE.69.066138)). (S,M) latent dimensions corresponds to (16D, 128D) in PTB and (32D, 512D) in Yelp15 and Yahoo Answers. For comparison, columns N(d) gives the entropy of a standard Normal in Rd.
['Data', 'sMIM (S)', 'N( [ITALIC] S)', 'AE (S)', 'sMIM (M)', 'N( [ITALIC] M)', 'AE (M)']
It is clear that sMIM learns a low entropy representation (i.e., lower than the anchor P(z)), whereas AE has no notion of a latent distribution, leading to high information entropy in the hidden state (i.e., more uniformly distributed, with less structure).