RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
Table 3: Accuracy on the Spider development and test sets, by difficulty as defined by Yu et al. (2018b).
['[BOLD] Split', '[BOLD] Easy', '[BOLD] Medium', '[BOLD] Hard', '[BOLD] Extra Hard', '[BOLD] All']
As expected, performance drops with increasing difficulty. The overall generalization gap between development and test of RAT-SQL was strongly affected by the significant drop in accuracy (9%) on the extra hard questions. When RAT-SQL is augmented with BERT, the generalization gaps of most difficulties are reduced.