Model-based Interactive Semantic Parsing:A Unified Framework and A Text-to-SQL Case Study
Table 3: Portion of interaction questions on right predictions (Qr%) for each agent setting on WikiSQL Dev set (smaller is better). “p∗/s∗=X” denotes our agent with probability/dropout-based error detection (threshold at X).
['[BOLD] SQLNet  [BOLD] System', '[BOLD] SQLNet  [BOLD] Avg. #q', '[BOLD] SQLNet  [BOLD] Qr%', '[BOLD] SQLova  [BOLD] System', '[BOLD] SQLova  [BOLD] Avg. #q', '[BOLD] SQLova  [BOLD] Qr%']
To better understand how each kind of error detectors works, we investigate the portion of questions that each detector spends on right predictions (denoted as “Qr”). An ideal system should ask fewer questions on right predictions while identify more truly incorrect predictions to fix the mistakes. One important conclusion drawn from this table is that probability-based error detection is much more effective on identifying incorrect predictions. Consider the system using probability threshold 0.5 for error detection (i.e., “p∗=0.5”) and the one using dropout-based error detector with a threshold 0.2 (i.e., “s∗=0.2”) on SQLNet. When both systems ask around the same number of questions during the interaction, the former spends only 16.9% of unnecessary questions on correct predictions (Qr), while the latter asks twice amount of them (32.1%). Similar situations are also observed for SQLova. It is also notable that, when the probability threshold is lower (which results in a fewer total number of questions), the portion of questions on right actions drops significantly (e.g., from 23.0% to 16.9% when the threshold changes from 0.8 to 0.5 on SQLNet). However, this portion remains almost unchanged for dropout-based error detection.