Dependency-based Hybrid Trees for Semantic Parsing
Table 3: Performance comparison with state-of-the-art models on GeoQuery dataset. († represents the system is using lambda-calculus expressions as meaning representations.)
['Type', 'System/Model', 'English (en)  [ITALIC] Acc.', 'English (en)  [ITALIC] F.', 'Thai (th)  [ITALIC] Acc.', 'Thai (th)  [ITALIC] F.', 'German (de)  [ITALIC] Acc.', 'German (de)  [ITALIC] F.', 'Greek (el)  [ITALIC] Acc.', 'Greek (el)  [ITALIC] F.', 'Chinese (zh)  [ITALIC] Acc.', 'Chinese (zh)  [ITALIC] F.', 'Indonesian (id)  [ITALIC] Acc.', 'Indonesian (id)  [ITALIC] F.', 'Swedish (sv)  [ITALIC] Acc.', 'Swedish (sv)  [ITALIC] F.', 'Farsi (fa)  [ITALIC] Acc.', 'Farsi (fa)  [ITALIC] F.']
Our model DepHT achieves competitive performance and outperforms the previous best system RHT on 6 languages. Improvements on the Indonesian dataset are particularly striking (+11.8 absolute points in F1). We further investigated the outputs from both systems on Indonesian by doing error analysis. We found 40 instances that are incorrectly predicted by RHT are correctly predicted by DepHT. We found that 77.5% of the errors are due to incorrect alignment between words and semantic units. We can see the question is asking “What state is San Antonio located in?”. However, the natural language word order in Indonesian is different from English, where the phrase “berada di” that corresponds to m2 (i.e., loc) appears between “San Antonio” (which corresponds to m5 – ′san antonio′) and “what” (which corresponds to m1 – answer). Such a structural non isomorphism issue between the sentence and the semantic tree makes the relaxed hybrid tree parser unable to produce a joint representation with valid word-semantics alignment. This issue makes the RHT model unable to predict the semantic unit m2 (i.e., loc) as RHT has to align the words “San Antonio” which should be aligned to m5 before aligning “berada di”. However, m5 has arity 0 and cannot have m2 as its child. Thus, it would be impossible for the RHT model to predict such a meaning representation as output. In contrast, we can see that our dependency-based hybrid tree representation appears to be more flexible in handling such cases. The dependency between the two words “di” (in) and “berada” (located) is also well captured by the arc between them that is labeled with m2. The error analysis reveals the flexibility of our joint representation in different languages in terms of the word ordering, indicating that the novel dependency-based joint representation is more robust and suffers less from language-specific characteristics associated with the data.