The Neural Noisy Channel
Table 1: ROUGE F1 scores on the sentence summarisation test set. The ‘uni’ and ‘bi’ in the parentheses denote the encoder for the model proposing candidates is a unidirectional LSTM or bidirectional LSTM. Those rows marked with an ∗ denote models that process their input online.
['Model', '# Parallel data', '# Data for LM', 'RG-1', 'RG-2', 'RG-L']
The noisy channel model with the language model trained on the target side of the 1 million parallel data outperforms the direct model by approximately 1 point. Such improvement indicates that the language model helps improve the quality of the output sequence when no extra unlabelled data is available. Training the language model with all the headlines in the dataset, i.e. 3.8 million sentences, gives a further boost to the ROUGE score. This is in line with our expectation that the model benefits from adding large amounts of unlabelled data. The interpolation of the direct model, channel model, language model and bias of the output length achieves the best results — the ROUGE score is close to the direct model trained on all the parallel data. Although there is still improvement, when the direct model is trained with more data, the gap between the direct model and the noisy channel model is smaller. No gains is observed if the language model is combined with the direct model. We find that as we increase the weight of the language model, the result is getting worse.