Learning an Executable Neural Semantic Parser
Table 8: Breakdown of questions answered by type for the GraphQuestions.
['Question type', 'Number', '% Answerable', '% Correctly answered']
GraphQuestions consists of four types of questions. An example of a relational question is what periodic table block contains oxygen; the second type contains count questions (denoted by count). An example is how many firefighters does the new york city fire department have; the third type includes aggregation questions requiring argmax or argmin (denoted by aggregation). An example is what human stampede injured the most people; the last type are filter questions which requires comparisons by >, ≥, < and ≤ (denoted by filter). An example is which presidents of the united states weigh not less than 80.0 kg. As the results reveal, relation questions are the simplest to answer which is expected since relation questions are non-compositional and their logical forms are easy to find by beam search. The remaining types of questions are rather difficult to answer: although the system is able to discover logical forms that lead to the correct denotation during beam search, the ranker is not able to identify the right logical forms to execute. Aside from the compositional nature of these questions which makes them hard to answer, another difficulty is that such questions are a minority in the dataset posing a learning challenge for the ranker to identify them. As future work, we plan to train separate rankers for different question types.