The Galactic Dependencies Treebanks: Getting More Data by Synthesizing New Languages
Table 3: Our final comparison on the 17 test languages appears in the upper part of this table. We ask whether single-source transfer to these 17 real target languages is improved by augmenting the source pool of 10 real languages with 1200 synthetic languages. When different languages are selected in these two settings, we boldface the setting with higher test UAS, or both settings if they are not significantly different (paired permutation test by sentence, p<0.05). For completeness, we extend the table with the 10 development languages. The “Avg.” lines report the average of 17 test or 27 test+dev languages. The two supervised-selection averages are significantly different (paired permutation test by language, p<0.05).
['target', 'unsupervised real', 'unsupervised +synthetic', '(weakly) supervised real', '(weakly) supervised +synthetic']
The above graphs are evaluated on development sentences in development languages. The comparison is similar to the comparison in the selection graph: do the synthetic treebanks add value? We use our largest source pools, n=10 and m=1210. With supervised selection, selecting the source language from the full pool of m options (not just the n real languages) tends to achieve significantly better UAS on the target language, often dramatically so. On average, the UAS on the test languages increases by 2.3 percentage points, and this increase is statistically significant across these 17 data points. Even with unsupervised selection, UAS still increases by 1.2 points on average, but this difference could be a chance effect.