Distant Supervision from Disparate Sourcesfor Low-Resource Part-of-Speech Tagging
Table 1: Results on the development sets and comparison of our best model to prior work. Lex: Size (word types) of dictionaries (W: Wiktionary, U: UniMorph). TCW: type-constraints using Wiktionary; →eW (embedded Wiktionary tags), DsDs: our model with →eW∪U. Results indicated by † use W only. Best result in boldface; in case of equal means, the one with lower std is boldfaced. Averages over language families (with two or more languages in the sample, number of languages in parenthesis).
['Language', 'Lex (103) W', 'Lex (103) U', 'Dev sets (UD2.1) 5k', 'Dev sets (UD2.1) TC [ITALIC] W', 'Dev sets (UD2.1)  [ITALIC] n-hot [ITALIC] W', 'Dev sets (UD2.1) → [ITALIC] eW', 'Dev sets (UD2.1) DsDs', '[EMPTY]', 'Test sets Das', 'Test sets Li', 'Test sets Garrette', 'Test sets Agic', 'Test sets DsDs']
We evaluate two dictionary sources, motivated by ease of accessibility to many languages: Wiktionary, a word type dictionary that maps tokens to one of the 12 Universal POS tags li-et-al:2012; petrov:ea:2012; and UniMorph, a morphological dictionary that provides inflectional paradigms across 350 languages (KIROV16.1077). For Wiktionary, we use the freely available dictionaries from \newciteli-et-al:2012 and \newciteagic:ea:2017. The size of the dictionaries ranges from a few thousands (e.g., Hindi and Bulgarian) to 2M (Finnish UniMorph). UniMorph covers between 8-38 morphological properties (for English and Finnish, respectively). There are several take-aways. The first take-away is that coverage-based instance selection yields substantially better training data. Training on 5k instances results in a sweet spot; more data (10k) starts to decrease performance, at a cost of runtime. Training on all WTC data (around 120k) is worse for most languages. We show: i) Li peaks at 10 iterations for their test languages, and at 35 iterations for all the rest. This is in slight contrast to 50 iterations that li-et-al:2012 recommend, although selecting 50 does not dramatically hurt the scores; ii) Our replication falls ∼5 points short of their 84.9 accuracy. There is a large 33-point accuracy gap between the scores of li-et-al:2012, where the dictionaries are large, and the other languages in