Unnatural Language Processing: Bridging the Gap Between Synthetic and Natural Language Data
Table 1: Overnight semantic parsing logical form accuracies on human-generated questions for models trained with synthetic data, real data, or both. Bold values indicate the best-performing model under the synth data condition, while underlined values indicate the best-performing model under any data condition. The projection approach consistently outperforms other models in the synthetic-only condition, and is comparable to a fully-supervised model in three domains.
['Data', 'Model', 'basketball', 'blocks', 'calendar', 'housing', 'pubs', 'recipes', 'restaurants', 'social', 'mean']
We report logical form exact match accuracies for the eight Overnight datasets. Three data conditions are evaluated: synth, which uses synthetic-only training data, real, which uses only human-annotated training data, and both, which combines the two. Human annotations are used for the test evaluation in all data conditions. For the both condition, we found that simply concatenating the two datasets was reliably better than any fine-tuning scheme.