Drinking from a Firehose: Continual Learning with Web-scale Natural Language
Table 6: Effect of K. Word perplexity of test performance (↓). ConGraD outperforms Online GD for all K. The results indicate that setting K adaptively is beneficial.
['[EMPTY]', '[ITALIC] K=1', 'Online GD  [ITALIC] K=3', 'Online GD  [ITALIC] K=5', 'ConGraD  [ITALIC] K=3', 'ConGraD  [ITALIC] K=5']
ConGraD outperforms the baselines for all K. For Online GD , K is the number of GD steps. The results indicate that different models and data call for different settings of K. For ConGraD, K is the maximum number of steps.