Localization of Fake News Detection via Multitask Transfer Learning
Table 6: An ablation study on the effect of multiple heads in the attention mechanisms. The results show that increasing the number of heads improves performance, though this plateaus at 10 attention heads. All ablations use the multitask-based finetuning method. “Effect” refers to the increase or decrease of accuracy as the heads are removed. Note that 10 heads is the default used throughout the study.
['# of Heads', 'Accuracy', 'Val. Loss', 'Effect']
As shown in Table 6, reducing the number of attention heads severely decreases multitasking performance. Using only one attention head, thereby attending to only one context position at once, degrades the performance to less than the performance of 10 heads using the standard finetuning scheme. This shows that more attention heads, thereby attending to multiple different contexts at once, is important to boosting performance to state-of-the-art results.