Latent Alignment and Variational Attention
Table 2: (Left) Performance change on NMT from exact decoding to K-Max decoding with K=5. (see section 5 for definition of K-max decoding). (Right) Test perplexity of different approaches while varying K to estimate Ez[p(y|x,~x)]. Dotted lines compare soft baseline and variational with full enumeration.
['Model', 'PPL Exact', 'PPL  [ITALIC] K-Max', 'BLEU Exact', 'BLEU  [ITALIC] K-Max']
For all methods exact enumeration is better, however K-max is a reasonable approximation. This possibly indicates that soft attention models are approximating latent alignment models. On the other hand, training with latent alignments and testing with soft attention performed badly.