LAMOL: LAnguage MOdeling for Lifelong Language Learning
Table 4: Summary of averaged score on five tasks. The scores are reported as the averaged score over all tasks of the models after training on every task. The rightmost three columns – LAMOL with γ=0.05 and γ=0.2 of real samples from previous tasks and Multitasked – are upper bounds for comparison. Best performance in boldface.
['Fine-tuned', 'MAS', 'LAMOL0.05GEN', 'LAMOL0.2GEN', 'LAMOL0.05TASK', 'LAMOL0.2TASK', 'LAMOL0.05REAL', 'LAMOL0.2REAL', 'Multitasked']
Also, as expected, the performance of LAMOL improves as the sampling ratio γ increases and task-specific tokens are used.