A Mixture of h-1 Heads is Better than h Heads
Table 7: IWSLT14 development set performance of different finetuning methods (§5.2). The last two columns indicate the number of parameters to update, and the number of gradient steps needed to achieve the best development performance.
['[BOLD] Method', '[BOLD] BLEU', '[BOLD] # Params.', '[BOLD] # Steps.']
Surprisingly, NoFt already outperforms Scratch without any finetuning. We attribute this improvement to the larger pretraining (WMT14) data. Only updating the gating functions, FtG improves over NoFt by 0.8 BLEU. Yet there is still a significant gap of 1.8 BLEU between FtG and FtAll. Interestingly, FtG+ almost matches the performance of FtAll, but only updates 1/9 as many parameters. Both FtG and FtG+ reach the best performance after around 1K gradient updates, i.e., one epoch, significantly less than FtAll or Scratch.