Fine-Grained Prediction of Syntactic Typology: Discovering Latent Structure with Supervised Learning
Table 5: Cross-validation average expected loss of the two grammar induction methods, MS13 (Mareček and Straka, 2013) and N10 (Naseem et al., 2010), compared to the EC heuristic of §5 and our architecture of §6 (the version from the last line of Table 3). In these experiments, the dependency relation types are ordered POS pairs. N10 harnesses prior linguistic knowledge, but its improvement upon MS13 is not statistically significant. Both grammar induction systems are significantly worse than the rest of the systems, including even our two baseline systems, namely EC (the “expected count” heuristic from §5) and ∅ (the no-feature baseline system from Table 4 line 0). Like N10, these baselines make use of some cross-linguistic knowledge, which they extract in different ways from the training treebanks. Among our own 4 systems, EC is significantly worse than all others, and +GD is significantly better than all others. (Note: When training the baselines, we found that including the +GD languages—a bias-variance tradeoff— harmed EC but helped ∅. The table reports the better result in each case.)
['[EMPTY]', 'MS13', 'N10', 'EC', '∅', 'UD', '+GD']
However, it is not a trained heuristic: it has no free parameters that we can tune to optimize our evaluation metric. For example, it can pay too much attention to tag pairs that are not discriminative. We therefore proceed to build a trainable, feature-based system. Even our baseline systems are significantly more accurate than the grammar induction systems, showing the effectiveness of casting the problem as supervised prediction.