Weakly Supervised POS TaggersPerform Poorly on Truly Low-Resource Languages
Table 2: POS tagging accuracy. For all neural models, the standard deviation is given in parentheses.
['[BOLD] Language am', '[BOLD] CHR11  [BOLD] 0.3441', '[BOLD] GAR13 0.1392', '[BOLD] PLA16 0.1643', '[BOLD] PLA16 (0.00)', '[BOLD] AMB 0.1595', '[BOLD] AMB (0.03)', '[BOLD] FREQ 0.2479', '[BOLD] FREQ (0.00)', '[BOLD] AMB+AE 0.1651', '[BOLD] AMB+AE (0.00)', '[BOLD] FREQ+AE 0.2608', '[BOLD] FREQ+AE (0.01)']
Our most important observation is that none of the approaches perform well on these languages. In fact, on average, no method gets even half of the tags right. This shows there is still a lot of room for exploring new and radically different approaches to learning POS taggers for (truly) low-resource languages. In addition, we make the following four observations: (i) We see that the unsupervised baseline, CHR11, obtains the lowest accuracy on average over all languages. This shows that, while all POS taggers are poor, distant supervision from tag dictionaries may provide some signal. (ii) On average over all languages, both single-task cross-lingual neural approaches, i.e., AMB and FREQ, outperform all baselines. Further, the difference between AMB and FREQ is with 0.0032 small. Thus, it seems that both cross-lingual strategies, i.e., ambiguous labeling and using the most frequent tag as the gold label, work similarly well on average. Looking at individual languages, however, up to around 0.1 performance difference can be found for am, hsb, hy, kmr, and te. Most likely, this difference might be explained by the frequency difference between the most frequent and other possible tags in the respective languages. (iii) Looking at PLA16, we find that a language modeling auxiliary task seems to hurt performance on average. PLA16 uses the ambiguous annotation strategy for cross-lingual transfer and should, thus, directly improve over AMB. However, it only performs better for three languages: am, ta, and tl. This contrast to \citeauthorPlank16multi \shortcitePlank16multi’s results could be due to the small corpus sizes and, thus, mistakes in estimating word frequencies. (iv) The approaches with character-level MTL, i.e., AMB+AE and FREQ+AE, both improve over the respective single-task approaches. AMB+AE also obtains the highest average accuracy overall. Thus, in contrast to the language modeling auxiliary task, a character-level multi-task approach improves model performance in our setting.