Identifying Domain Adjacent Instances for Semantic Parsers
Table 4: Accuracy for a semantic parser evaluated on a test set in which 20% is domain adjacent.
['[EMPTY]', 'Basketball', 'Blocks', 'Calendar', 'Housing', 'Publications', 'Recipes', 'Restaurants', 'Social']
In general, the relative performance is similar to that in the direct evaluation (e.g. Surprise tends to do well on most domains, but performs poorly on Basketball and Social in particular). However, in this evaluation, misclassifying an instance as domain-adjacent if the semantic parser would have accurately parsed it is worse than misclassifying the instance if the semantic parser could not have accurately parsed it. For example, in Social we can thus infer that Surprise is marking some instances as domain-adjacent that would otherwise be accurately parsed as the performance there is actually worse than for NoFilter.