Syntactic Question Abstraction and Retrieval for Data-Scarce Semantic Parsing
Table 4: Comparison of model with three WikiSQL train subsets: Train-Rand-2677, Train-Uniform-85P-2550 and Train-Hybrid-96P-2750).
['Model', 'Train set', 'Dev set', 'P (%)', 'LF (%)']
To further test generality of our findings under change of train set size Train-Uniform-85P-2550 consists of 2550 uniformly distributed examples over 85 patterns, Train-Rand-2677 consists of 2667 examples randomly sampled from the WikiSQL train data, and Train-Hybrid-96P-2750 is larger version of Train-Hybrid-85P-897 in which each logical pattern includes at least 15 examples for 96 logical patterns (Table. (1) SQAR shows marginally better performance than SQLova showing +1.9%, +0.5%, and -0.7% in LF when Train-Rand-2677, Train-Uniform-85P-2550, and Train-Hybrid-96P-2750 are used as the train sets (1st and 3rd rows of each panel); (2) Again, the pre-training using Quora paraphrasing datset increases LF by +0.5%, +3.3%, and +2.7% in Train-Rand-2677, Train-Uniform-85P-2550, and Train-Hybrid-96P-2750 respectively (2nd and 3th rows of each panel); (3) Both SQAR and SQLova show best performance when they are trained over hybrid dataset. Observing that the performance gap between SQAR and SQLova becomes marginal as increasing the size of train set, we train both models using full WikiSQL train set. The result shows that again, there is only marginal difference between two models (SQLova LF: 79.2±0.1, SQAR LF = 78.4±0.2). The overall results are summarized in Fig.