Syntactic Question Abstraction and Retrieval for Data-Scarce Semantic Parsing
Table 5: Parsing unseen logical forms. SQAR is trained by using Train-Rand-881 and P and LF are measured while using a different set for query retrieval in the inference stage. R-881, H-897, H-2750, and Full stand for Train-Rand-881, Train-Hybrid-85P-897, Train-Hybrid-96P-2750, and Train-Full-56355 respectively. R-capacity indicates the number of successfully retrieved logical pattern types whereas RG-capacity indicates that of successfully parsed logical pattern types.
['Model', 'Train set', 'Set for retrieval', 'P (%)', 'LF (%)', 'R-capacity', 'RG-capacity']
To experimentally show this, we measured P and LF of SQAR while changing the retrieval set during an inference stage (Table. The train set is fixed to Train-Rand-881 consisting of 67 logical patterns. The result shows that upon addition of Train-Hybrid-85P-897 into the template set, which includes 18 more logical patterns compared to Train-Rand-881, P and LF increases by 1.1% and 0.6% respectively (2nd row of the table). Similar results are observed with Train-Hybrid-96P-2750 (+2.0% in P and +0.7% LF, 3rd row of the table) and with Test-Full-15878 (+4.1% in P and +1.7% in LF, 4th row of the table). To further show the power of using query-to-query similarity, we replaced the entire retrieval set from Train-Rand-881 to Train-Hybrid-96P-2750 where only 43 examples are overlapped between them. Again, P and LF increase by 1.7% and 0.5% respectively (5th row of the table). To further confirm the addition of examples enables parsing of unseen logical patterns, we introduce two additional metrics: R-capacity and RG-capacity. R-capacity is defined by the number of successfully retrieved logical pattern types by SQAR in the test set whereas RG-capacity indicates the number of successfully generated (retrieved and grounded) logical pattern types. The table shows both R- and RG-capacities increases upon addition of examples into the retrieval set (5th and 6th columns). It should be emphasized that, during the training stage, SQAR observed only 67 logical patterns. Collectively, these results show that, SQAR can be easily generalized to handle new logical patterns by simply adding new examples without re-training. This also shows the possibility of transfer learning, even between semantic parsing tasks using different logical forms as intermediate logical patterns can be obtained from the natural language space.