Drinking from a Firehose: Continual Learning with Web-scale Natural Language
Table 7: Validation buffer size. Word perplexity (↓) of Mixed Replay × ConGraD.
['| [BOLD] V|=90 [ITALIC] K', '| [BOLD] V|=180 [ITALIC] K', '| [BOLD] V|=270 [ITALIC] K', '| [BOLD] V|=360 [ITALIC] K', '| [BOLD] V|=450 [ITALIC] K']
Online validation buffer size. We use Mixed Replay × ConGraD for this experiment. The results indicate that a small validation buffer (two data points per user) is sufficient to successfully adapt gradient descent. Although one would expect better performance with a bigger validation set, we see lower performance with large buffer sizes. We hypothesize that larger validation buffer size hurts learning because it creates a delay between the time data is observed and the time it is used for learning.