Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
Table 11: Empirical expected pairwise KLD on PTB.
['[BOLD] Model', '[BOLD] Validation', '[BOLD] Test']
What’s more, another indicator of high rank is that the model can precisely capture the nuance of difference contexts. If a model can better capture the distinctions among contexts, we expect the next-step conditional distributions to be less similar to each on average. Based on this intuition, we use the expected pairwise Kullback–Leibler divergence (KLD), i.e., Ec,c′∼C[KLD(P(X∣c)∥P(X∣c′))] where C denotes all possible contexts, as another metric to evaluate the ranks of the three models (MoS, MoC and Softmax). As we expected, MoS achieves higher expected pairwise KLD, indicating its superiority in covering more contexts of the next-token distribution.