Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling
Table 8: Same as Table 7, except filtering for word types that occur fewer than 5 times in the training set. The cache component is used as expected even on rare words: proper nouns are extremely likely to have been cache-generated, whereas numbers and generic content words are less likely to have been; this indicates both the effectiveness of the prior at determining whether to use the cache and the burstiness of proper nouns.
['Word Gore', '¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ [ITALIC] p( [ITALIC] z∣ [ITALIC]  [BOLD] w)↓ 0.977', 'Word 770', '¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ [ITALIC] p( [ITALIC] z∣ [ITALIC] w)↑ 0.003']
Tab. The pattern here is similar to the unfiltered list: proper nouns are extremely likely to have been cache-generated, whereas numbers and generic (albeit infrequent ) content words are less likely to have been.