Revisiting Self-Trainingfor Neural Sequence Generation
Table 3: Results on the toy sum dataset. For ST and noisy ST, smoothness (↓) and symmetric (↓) results are from the pseudo-training step, while test errors (↓) are from fine-tuning, all at the first iteration.
['Methods', 'smoothness', 'symmetric', 'error']
For any integer pair (x1,x2), we measure local smoothness as the standard deviation of the predictions in a 3×3 neighborhood of (x1,x2). These values are averaged over all the 10000 points to obtain the overall smoothness. To demonstrate the effect of smoothing on the fine-tuning step, we also report test errors after fine-tuning. We observe that ST pseudo-training attains better smoothness, which helps reducing test errors in the subsequent fine-tuning step. If we perturb the input during the pseudo-training step, then Eq. (3) where g(x) is a perturbation function. We first validate noisy ST in the toy sum task. We shuffle the two integers in the input as the perturbation function. Such perturbation is suitable for this task since it would help the model learn the commutative law as well. To check that, we also measure the symmetry of the output space. Specifically, for any point (x1,x2), we compute |f(x1,x2)−f(x2,x1)| and average it over all the points. While we do not explicitly perturb the input at nearby integers, the shuffling perturbation greatly improves the smoothness metric as well. Furthermore, predictions are more symmetric and test errors are reduced.