CUED@WMT19:EWC&LMs
Table 4: Fine-tuning our models on former WMT test sets using continued training and EWC.
['[EMPTY]', '[BOLD] Fine-tuning', '[BOLD] Checkpoint  [BOLD] averaging', '[BOLD] BLEU (test18)  [BOLD] En-De', '[BOLD] BLEU (test18)  [BOLD] De-En']
Similarly to Schamper et al. Due to the very small size of corpus B, much care has to be taken to avoid over-fitting. We experimented with different techniques that keep the model parameters in the fine-tuning phase close to the original ones. First, we fine-tuned our models for about 1K-2K iterations (depending on the performance on the news-test2017 dev set) and dumped checkpoints every 500 steps. Averaging all fine-tuning checkpoints together with the last unadapted checkpoint yields minor gains over fine-tuning without averaging (rows 3 vs. 4 in Tab. – that explicitly penalizes the distance of the model parameters θ to the optimized but unadapted model parameters θ∗A. The regularized training objective according EWC is: L(θ)=LB(θ)+λ∑iFi(θi−θ∗A,i)2 (1) where LB(θ) is the normal cross-entropy training loss on task B and Fi=E[∇2LA(θi)] is an estimate of task A Fisher information, which represents the importance of parameter θi to A. On English-German, fine-tuning with EWC and checkpoint averaging yields an 1.1 BLEU improvement (rows 1 vs. 6 in Tab. Gains are generally smaller on German-English.