Machine Translation System Selection from Bandit Feedback
Table 4: Effects of simulated feedback method on performance.
['[EMPTY]', 'R', 'B', 'T']
We find that the nature of feedback has little overall effect on system performance. Perturbing the loss to model Skew was the notably detrimental, but the remaining perturbations performed similarly. Surprisingly we observe no significant effect moving from Scale, which is essentially the full continuous BLEU metric, to coarser or more distorted variants.