Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection
Table 6: Nested datasets: Data order is important for denoising. S80%⊃S40%⊃S20% with stricter/smaller set less noisy.
['[EMPTY]', 'Subset', 'n2014', 'd2015', 'patent']
In another experiment, we select 3 data subsets based on the amount of noise in each sentence pair, each subset being noise-reduced to different degree. For example, we select top 80% least noisy sentence pairs (denoted as S80%) out the entire Paracrawl data. Then we select the top half of S80% which is essentially 40% of the Paracrawl data. We denote it as S40%, similarly, S20%, therefore S80%⊃S40%⊃S20%. Then we fine-tune P1 (noisy baseline) on S80% with early stopping on devset, followed by the fine-tuning on S40% and then S20%. This also confirms the importance of the right data order in denoising.