LAMOL: LAnguage MOdeling for Lifelong Language Learning
Table 3: Summary of averaged metric scores for different methods under permuted task orders using models at last epoch of last task. The Average and Std columns respectively are the average and standard deviation of the averaged scores for each row of the methods. Multitasked learning as an upper bound is shown at the bottom.
['Methods', 'SST SRL WOZ', 'SST WOZ SRL', 'SRL SST WOZ', 'SRL WOZ SST', 'WOZ SST SRL', 'WOZ SRL SST', 'Average', 'Std']
For an initial understanding of the performance on all of the methods and the effect of task order, we first conducted a small-scale experiment on three small datasets: SST, QA-SRL, and WOZ. We trained all but the the multitasked method on all six permutations of the task order. The final score for each order was obtained by evaluating the model at the conclusion of the training process. Note that LAMOL with γ=0 is not the same as Fine-tuned, as the LM loss is still optimized. [leftmargin=*] Fine-tuned, EWC, MAS, and LAMOL with γ=0 show similar performance and are much worse than LAMOL with γ>0. LAMOL0.2GEN, our best performing method, is only 1.8 percent away from Multitasked, which implies almost no forgetting during LLL. The order of the tasks is crucial to the performance. For instance, the WOZ score drops significantly after training other tasks. Thus, if WOZ is not the last task, the performance is usually noticeably worse. When using LAMOL, the performance of old tasks maintains almost the same level throughout the training process. When the sampling ratio γ is increased, the performance also increases, especially when increased from 0 to 0.05. When γ=0, adding task-specific tokens harms performance, because the model must fit additional special tokens that are useless. Adding task-specific tokens is also not helpful if γ=0.2. We believe that 0.2 is enough for three tasks; thus task-specific tokens are redundant. However, when γ=0.05, task-specific tokens are beneficial because the tokens are needed to help retain a substantial presence of the first task when training the third task. We see that a better LLL method usually has a smaller standard deviation, which implies that it is effected less by task order. Adding task-specific tokens also has a stabilizing effect.