Don’t paraphrase, detect! Rapid and Effective Data Collection for Semantic Parsing
Table 3: Denotation accuracy for the set of examples in Dnat covered by Don (acccov), and for the set of uncovered examples (accdisj).
['Dataset', '[ITALIC] acccov', '[ITALIC] accdisj']
To verify whether coverage is indeed important in our setup, we performed the following experiment. Then, we calculated the accuracy of the model with respect to examples for which their template appears in Don (acccov), and for those their template does not appear (accdisj). This shows the importance of generating logical form templates that are likely to appear in the target distribution. Thus, our finding reinforces the conclusions in finegan-dollak-etal-2018-improving, and shows that while neural semantic parsers obtain good performance on existing benchmarks, their generalization to new compositional structures is limited.