SParC: Cross-Domain Semantic Parsing in Context
Table 8: Performance stratified by question difficulty on the development set. The performances of the two models decrease as questions are more difficult.
['Goal Difficulty', 'CD-Seq2Seq', 'SyntaxSQL-con']
We group individual questions in SParC into different difficulty levels based on the complexity of their corresponding SQL representations using the criteria proposed in Yu et al. As shown in Figure 3, the questions turned to get harder as interaction proceeds, more questions with hard and extra hard difficulties appear in late turns. As we expect, the models perform better when the user request is easy. Both models fail on most hard and extra hard questions.