Parser Training with Heterogeneous Treebanks
Table 1: LAS scores when testing on the training treebank and on the PUD test set with training treebank as proxy. For each test set, the best result is marked with bold. Treebank size is given as number of sentences in the training data. Statistically significant differences, at the 0.05-level, from single are marked with +, from concat with × and from both these systems with *. For clarity, significance for PUD is only shown for the proxy treebank with the highest score.
['[BOLD] Language', '[BOLD] Treebank', '[BOLD] Size', 'Same treebank test set single', 'Same treebank test set concat', 'Same treebank test set c+ft', 'Same treebank test set tb-emb', 'PUD test set single', 'PUD test set concat', 'PUD test set c+ft', 'PUD test set tb-emb']
We performed all experiments on UD version 2.1 treebanks Nivre et al. We selected 9 languages, based on the criteria that they should have at least two treebanks with fully available training data and a PUD test set. The situation is quite different across languages with either treebanks of roughly the same size, as for Spanish, or very skewed data sizes with a mix of large and small treebanks, as for Czech. In all cases we use all available data, except for Czech, where we randomly choose a maximum of 15,000 sentences per treebank per epoch for efficiency reasons. Overall we observe substantial gains when using either c+ft or tb-emb. On average both c+ft and tb-emb beat single by 3.5 LAS points and concat by over 2.0 LAS points when testing on the test sets of the treebanks used for training, and both methods beat both baselines by over 2.0 LAS points for the PUD test set, if we consider the best proxy treebank.