What to do about non-standard (or non-canonical) language in NLP
Table 2: Tagging accuracy on various test set varieties (domains, languages and age groups; Tw=Twitter), using coarse POS [petrov:ea:2012]. OOV: out-of-vocabulary rate wrt Wsj.train. Accuracy is significantly correlated with OOV rate (ρ = -0.70).
['Variety', 'Sample', 'TnT', 'Bilty→ [ITALIC] w', 'Bilty→ [ITALIC] w+→ [ITALIC] c', 'OOV']
First, as is well known, we see that all taggers suffer when applied to other domains. However, models trained on WSJ fare worse on data from the younger age group, thus age is a covariate. This confirms the age bias reported in [hovy:soegaard:2015:age] for the same data but using different taggers. If we stretch the notion of variety to other languages, we see that performance unsurprisingly drops dramatically. Remember, we just apply an in-domain single language tagger to other languages, although only trained on WSJ here.