Localization of Fake News Detection via Multitask Transfer Learning
Table 5: An ablation study on the effects of pretraining for multitasking-based and standard GPT-2 finetuning. Results show that pretraining greatly accounts for almost half of performance on both finetuning techniques. “Acc. Inc.” refers to the boost in performance contributed by the pretraining step. “% of Perf.” refers to the percentage of the total performance that the pretraining step contributes.
['Finetuning', 'Pretrained?', 'Accuracy', 'Val. Loss', 'Acc. Inc.', '% of Perf.']
In Table 5, it can be seen that generative pretraining via language modeling does account for a considerable amount of performance, constituting 44.32% of the overall performance (a boost of 42.67% in accuracy) in the multitasking setup, and constituting 43.93% of the overall performance (a boost of 39.97%) in the standard finetuning setup.