Error-Correcting Neural Sequence Prediction
Table 1: LSTM Perplexities for Full Softmax (SM), Sample-Based SM (Sample-SM), Hierarchical-SM (HSM), Adaptive-SM, NCE and ECOC-NSP.
['[BOLD] Model', '[BOLD] PTB Val.', '[BOLD] PTB Test', '[BOLD] WikiText-2 Val.', '[BOLD] WikiText-2 Test', '[BOLD] WikiText-103 Val.', '[BOLD] WikiText-103 Test']
Baselines for ECOC-NSP For HS, we use a 2-hidden layer tree with a branching factor (number of classes) of √|V| by default. For AS, we split the output into 4 groups via the unigram distribution (in percentages of total words 5%-15%-30%-100%). For NCE, we set the noise ratio to be 0.1 for PTB and 0.2 for WikiText-2 and WikiText-103. Training is carried out until near convergence (ϵ≈40), the randomly initialized HS and Sampled Softmax of which take longer (ϵ∈ [55-80]). For Rand/Unigram Hierarchical SM, we use a 2-hidden layer tree with 10 classes per child node. almost performs as well as the full-softmax (8.02M parameters) while only using 1000 bits for PTB (|V|/20 and ) and 5K bits for WikiText-2 (|V|/25) and WikiText-103 (|V|/30). The HS-based models use a 2-hidden layer tree with 10 tokens per class, resulting in 4.4M parameters for PTB, 22.05M parameters for WikiText-2 (full softmax - 40.1M) and WikiText-103. Moreover, we find there is a consistent improvement in using Embedding-ECOC over using a random codebook (Random-ECOC) and a slight improvement over using a unigram ordered codebook (Uni-ECOC). We also found that too many bits e.g |C|=|V| takes much longer (ϵ∈ [20-30] more for PTB) to converge with negligible perplexity reductions. Hence, the main advantage of ECOC-NLVMS is the large compression rate while maintaining performance (e.g PTB with |C|=40, there is less than 2 perplexity points compared to the full softmax).