Adversarially Regularized Autoencoders
Figure 4: Reconstruction error (negative log-likelihood averaged over sentences) of the original sentence from a corrupted sentence. Here k is the number of swaps performed on the original sentence.
['[ITALIC] k', 'AE', 'ARAE']
We start with a simple proxy that a smooth encoder model should map similar sentences to similar z values. For 250 sentences, we calculate the average cosine similarity of 100 randomly-selected sentences within an edit-distance of at most 5 to the original. To further test this property, we feed noised discrete input to the encoder and (i) calculate the score given to the original input, and (ii) compare the resulting reconstructions. We observe that ARAE is able to map a noised sentence to a natural sentence (though not necessarily the denoised sentence). We obtain the reconstruction error (negative log likelihood) of the original non-noised sentence under the decoder, utilizing the noised code. We find that when k=0 (i.e. no swaps), the regular AE better reconstructs the exact input. However, as the number of swaps pushes the input further away, ARAE is more likely to produce the original sentence. (Note that unlike denoising autoencoders which require a domain-specific noising function