The Neural Noisy Channel
Table 4: BLEU scores from different models for the Chinese to English machine translation task.
['Model', 'BLEU']
To set benchmarks, we train the vanilla and attentional sequence to sequence models (sutskever:2014; bahdanau:2015) using the same parallel data. For direct models, we leverage bidirectional LSTMs as the encoder for this task. We can see that the vanilla sequence to sequence model behaves poorly due to the small amounts of parallel data. By contrast, the direct model (SSNT) and the attentional model work relatively well, with the attentional model outperforming the SSNT direct model. Although these models both directly model p(y∣x), this result is unsurprising because the SSNT direct model is most effective when the alignment between sequences is largely monotonic, and Chinese–English translation word orders diverge considerably. However, despite this limitation, the noisy channel model is approximately 3 points higher in BLEU than the direct model, and the combination of noisy channel and direct model gives extra boost. Confirming the empirical findings of prior work (and in line with theoretical predictions), the interpolation of the direct model and language model is not effective.