Drinking from a Firehose: Continual Learning with Web-scale Natural Language
Table 3: Word perplexity (↓) of a personalized vs a user-agnostic model (size in parentheses) on Firehose10M.
['[EMPTY]', 'User-agnostic (57.4M)', 'Personalized (65.1M)']
Multi-task structure. We conduct an experiment to evaluate the importance of multi-task modeling on Firehose. To this end, we examine whether a single language model can fit the data as well as a personalized language model. For this experiment, we trained the models offline, in a batch setting, to specifically evaluate the prevalence of multi-task structure in the data. The personalized model reduces word-level perplexity by 16.3% relative to the user-agnostic model. To further evaluate the multi-task structure, we evaluate each user’s language model on data from 5 other randomly chosen users. Cross-user perplexity is as high as 159.2. This indicates that tasks (users in our setting) differ significantly from each other, validating the multi-task structure.