Learning a Neural Semantic Parser from User Feedback
Table 7: Percentage of examples that required annotation (i.e., where the model initially made an incorrect prediction) on Geo880 vs. batch size.
['[BOLD] Batch Size', '150', '100', '50']
As in the live experiment, accuracy improves with successive batches. Data augmentation using templates helps in the initial stages of Geo880, but its advantage is reduced as more labeled data is obtained. Templates did not improve accuracy on ATIS, possibly because most ATIS queries involve two entities, i.e., a source city and a destination city, whereas our templates only generate questions with a single entity type. Nevertheless, templates are important in a live system to motivate users to interact with it in early stages. As observed before, paraphrasing improves performance at all stages. Smaller batch sizes reduce annotation effort, with a batch size of 50 requiring only 54.3% of the examples to be annotated. This result demonstrates that more frequent deployments of improved models leads to fewer mistakes.