Distilling Knowledge for Search-based Structured Prediction
Table 3: The machine translation results. MIXER denotes that of Ranzato et al. (2015), BSO denotes that of Wiseman and Rush (2016). Significance test Koehn (2004) shows the improvement of our Distill (both) over Baseline is statistically significant with p<0.01.
['[EMPTY]', 'BLEU']
Similar to the PTB parsing results, the ensemble 10 translators outperforms the baseline translator by 3.47 in BLEU score. Distilling from the ensemble by following the reference leads to a single translator of 24.76 BLEU score. We set T=0.1 in our distillation from exploration experiments since it achieves the best development score. Distilling from both the reference and exploration improves the single model’s performance by a large margin and achieves a BLEU score of 25.44.