Explaining Queries over Web Tables to Non-Experts
Table 9: Effect of user feedback on correctness
['[BOLD] train ex.', '[BOLD] annotations', '[BOLD] correctness', '[BOLD] MRR']
We experiment on two scenarios: (1) training the parser solely on 1,650 annotated examples; (2) integrating our training examples into the entire WikiTableQuestions training set of 11K examples. For each scenario we trained two parsers, one trained using annotations and the other without any use of annotations. To gain more robust results we ran our experiments on three different train/dev splits of our data, averaging the results. When training solely on the annotated examples, parser correctness on development examples increased by 8% (41.8% to 49.8%). The spike in correctness shows that feedback acquired using our explanations is high-quality input for the semantic parser, hence the parser achieves better correctness when trained on it compared to training on the original WikiTableQuestions benchmark.