Dialogue Learning With Human-in-the-Loop
Table 6: Second Iteration of Feedback Using synthetic textual feedback of synthetic Task2+3 with the RBI+FP method, an additional iteration of data collection of 10k examples, varying sparse binary reward fraction r and exploration ϵ. The performance of the first iteration model was 0.478.
['[EMPTY]', '[ITALIC] r=0', '[ITALIC] r=0.1', '[ITALIC] r=0.5', '[ITALIC] r=1']
We conducted experiments with an additional iteration of data collection for the case of binary rewards and textual feedback using the synthetic Task 2+3 mix. Using that model as a predictor, we collected an additional 10,000 training examples. We then continue to train our model using the original 1k+10k training set, plus the additional 10k. As before, we report the test accuracy varying r on the additional collected set. We also report the performance from varying ϵ, the proportion of random exploration of predictions on the new set. Overall, performance is improved in the second iteration, with slightly better performance for large r and ϵ=0.5. However, the improvement is mostly invariant to those parameters, likely because FP takes advantage of feedback from incorrect predictions in any case.