From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers
Table 4: Results of the few-shot experiments with varying numbers of target-language examples k. For each k, we report performance averaged across languages and the difference (Δ) with respect to the zero-shot setting.
['Task', 'Model', 'Sampling', '[ITALIC] k  [ITALIC] k=0', '[ITALIC] k=10 score', '[ITALIC] k=10 Δ', '[ITALIC] k=50 score', '[ITALIC] k=50 Δ', '[ITALIC] k=100 score', '[ITALIC] k=100 Δ', '[ITALIC] k=500 score', '[ITALIC] k=500 Δ', '[ITALIC] k=1000 score', '[ITALIC] k=1000 Δ']
We note substantial improvements in few-shot learning setups for all tasks. However, the results also reveal notable differences between different types of tasks. For higher-level language understanding tasks the improvements are less pronounced; the maximum gains after seeing k=1,000 target-language instances and 10 articles, respectively, are between 2.1 and 4.57 points. On the other hand, the gains for the lower-level tasks are massive: between 14.11 and 26 percentage points, choosing the best sampling strategy for each task. Moreover, the gains in all lower-level tasks are substantial even when we add only 10 annotated sentences in the target language (on average, up to 14.45 percentage points on DEP and 13.42 points on POS). What is more, additional experiments (omitted for brevity) show substantial gains for DEP and POS even when we add fewer than 5 annotated sentences. Part-of-Speech Tagging. Hovy et al. measure agreement of crowdsourced POS annotations with expert annotations ; they crowdsource annotations for 1,000 tweets, at a cost of $0.05 for every 10 tokens. With a total of 14,619 tokens in the corpus, this amounts to approximately $73 for 1,000 tweets, which is ≥1,000 sentences. Named Entity Recognition. Bontcheva et al. At a median pay of $11.37/hr, this amounts to around 190 sentences annotated in an hour. In other words, in less than 3 hours, we can collect more than 500 annotated examples.