Piecewise Latent Variables for Neural Variational Text Processing
Table 6: Approximate posterior word encodings (20-NG). For P-KL, we bold every case where piecewise variables showed greater word sensitivity than Gaussian variables w/in the same hybrid model.
['[BOLD] Word', '[BOLD] G-NVDM', '[BOLD] H-NVDM-5', '[EMPTY]']
The Gaussian variables were originally were sensitive to some of the words in the table. However, in the hybrid model, nearly all of the temporal words that the Gaussian variables were once more sensitive to now more strongly affect the piecewise variables, which themselves also capture all of the words that were originally missed This shift in responsibility indicates that the piecewise constant variables are better equipped to handle certain latent factors. This effect appears to be particularly strong in the case of certain nationality-based adjectives (e.g., “american”, “israeli”, etc.). While the G-NVDM could model multi-modality in the data to some degree, this work would be primarily done in the model’s decoder. In the H-NVDM, the piecewise variables provide an explicit mechanism for capturing modes in the unknown target distribution, so it makes sense that the model would learn to use the piecewise variables instead, thus freeing up the Gaussian variables to capture other aspects of the data, as we found was the case with names (e.g., “jesus”, “kent”, etc.).