Unnatural Language Processing: Bridging the Gap Between Synthetic and Natural Language Data
Table 2: Instruction following accuracies on the BabyAI SynthLoc task with human-generated instructions and models trained on synthetic or a mix of synthetic and real data. Bold values indicate the best-performing model for a given data condition and underlined values indicate the best-performing model under any data condition. Projection (α=0.1) gives the best results on both success (task completion) and discounted reward metrics, outperforming other models trained on only synthetic data as well as models fine-tuned on 2048 real examples.
['Data', 'Model', 'success', 'reward']
Again, projection is the best way to incorporate pretrained representations: introducing bert embeddings into the underlying sequence-to-sequence model makes performance worse. (We attribute this to overfitting to the limited set of word embeddings observed during training.) Again, as well, projection is effective enough to allow synthetic-only training to outperform fine-tuning on a small amount of human data. These results indicate that sim-to-real transfer is also an effective strategy for tasks whose sample complexity results from the difficulty of an underlying planning problem rather than language understanding as such.