FRAGE: Frequency-Agnostic Word Representation
Table 2: Perplexity on validation and test sets on Penn Treebank and WikiText2. Smaller the perplexity, better the result. Baseline results are obtained from DBLP:journals/corr/abs-1708-02182 ; DBLP:journals/corr/abs-1711-03953 . “Paras” denotes the number of model parameters.
['[EMPTY]', '[EMPTY]', '[BOLD] Paras', '[BOLD] Orig. Validation', '[BOLD] Orig. Test', '[BOLD] with FRAGE Validation', '[BOLD] with FRAGE Test']
Language Modeling In all these settings, our method outperforms the two baselines. On PTB dataset, our method improves the AWD-LSTM and AWD-LSTM-MoS baseline by 0.8/1.2/1.0 and 0.76/1.13/1.15 points in test set at different checkpoints. On WT2 dataset, which contains more rare words, our method achieves larger improvements. We improve the results of AWD-LSTM and AWD-LSTM-MoS by 2.3/2.4/2.7 and 1.15/1.72/1.54 in terms of test perplexity, respectively.