Don’t paraphrase, detect! Rapid and Effective Data Collection for Semantic Parsing
Table 2: Logical form coverage of Dnat by Don, after converting logical forms to their templates, for different values of D.
['Dataset', '[ITALIC] D', '|Don|', 'Dnat coverage']
Because Overnight requires paraphrasing each generated example, a reasonable value for D is 5. We observe that in this case only 63.73% of the examples in Scholar are covered, as well as 76.52% of the examples in GeoQuery. First, we generate all canonical utterances Con up to a depth D from the Overnight grammar. Because we do not paraphrase all the canonical utterances, we can generate to a higher depth D compared to Overnight and cover more of the examples in Xul Then, we iteratively label the utterances in Xul. At each iteration t, we use a paraphrase detection model st(x,c) to present promising canonical utterances to crowd workers for each unlabeled utterance x, who label the dataset. Importantly, crowd workers in our setup do not generate paraphrases, they only detect them. We now describe GrAnno in more detail.