Model-based Interactive Semantic Parsing:A Unified Framework and A Text-to-SQL Case Study
Table 2: Simulation evaluation of MISP-SQL (based on SQLNet or SQLova) on WikiSQL Test set. “MISP-SQLp∗=X” denotes our agent with probability-based error detection (threshold at X). “MISP-SQLUnlimitK” denotes a variant that asks questions for every component, with up to K+1 questions per component.
['[BOLD] System', '[BOLD] SQLNet  [BOLD] Accqm', '[BOLD] SQLNet  [BOLD] Accex', '[BOLD] SQLNet  [BOLD] Avg. #q', '[BOLD] SQLova  [BOLD] Accqm', '[BOLD] SQLova  [BOLD] Accex', '[BOLD] SQLova  [BOLD] Avg. #q']
For SQLNet, we also compare our system with the reported performance of DialSQL However, since DialSQL is not open-sourced and it is not easy to reproduce it, we are unable to adapt it to SQLova for more comparisons. Following Xu et al. ; Hwang et al. “Avg. #q” denotes the average number of questions per query. For any base parser, MISP-SQL improves their performance by interacting with users. Particularly for SQLNet, MISP-SQL outperforms the DialSQL system with only half the number of questions (1.104 vs. 2.4), and has a much simpler design without the need of training an extra model (besides training the base parser, which DialSQL needs to do as well). Our agent can even boost the strong performance of SQLova from 85% to 94% in execution accuracy, with merely 0.773 questions per query.