LAMOL: LAnguage MOdeling for Lifelong Language Learning
Table 2: Comparison of GPT-2 and other methods on single task scores. Other scores are retrieved from McCann2018decaNLP or d2019episodic. Better performance in boldface.
['[EMPTY]', 'SQuAD', 'WikiSQL', 'SST', 'SRL', 'WOZ', 'AGNews', 'Amazon', 'DBPedia', 'Yahoo', 'Yelp']
To establish a reference on the capability of the GPT-2 model on every dataset, we trained the model on each dataset independently. We observe that the performance of the GPT-2 model is actually quite good, even beating the BERT-based model (d2019episodic) on text classification datasets by a large margin. Thus, the GPT-2 model has the potential for superior LLL performance, as long as we can prevent catastrophic forgetting.