Backpropagating through Structured Argmax using a spigot
Table 3: Syntactic parsing performance (in unlabeled attachment score, UAS) and DM semantic parsing performance (in labeled F1) on different groups of the development data. Both systems predict the same syntactic parses for instances from Same, and they disagree on instances from Diff (§5).
['[BOLD] Split', '[BOLD] # Sent.', '[BOLD] Model', '[BOLD] UAS', '[BOLD] DM']
We consider the development set instances where both syntactic and semantic annotations are available, and partition them based on whether the two systems’ syntactic predictions agree (Same), or not (Diff). The second group includes sentences with much lower syntactic parsing accuracy (91.3 vs. 97.4 UAS), and spigot further reduces this to 89.6. Even though these changes hurt syntactic parsing accuracy, they lead to a 1.1% absolute gain in labeled F1 for semantic parsing. Furthermore, spigot has an overall less detrimental effect on the intermediate parser than ste: using spigot, intermediate dev. parsing UAS drops to 92.5 from the 92.9 pipelined performance, while ste reduces it to 91.8.