Learning to select data for transfer learning with Bayesian Optimization
Table 3: Results for data selection for part-of-speech tagging and parsing domain adaptation on the SANCL 2012 shared task dataset (Petrov and McDonald, 2012). POS: Part-of-speech tagging. Pars: Parsing. POS tagging models: Structured Perceptron (P); Bi-LSTM tagger (B) (Plank et al., 2016). Parsing model: Bi-LSTM parser (BIST) (Kiperwasser and Goldberg, 2016). Evaluation metrics: Accuracy (POS tagging); Labeled Attachment Score (parsing). Best: bold; second-best: underlined.
['[EMPTY]', '[BOLD] Trg domains →  [BOLD] Task →', '[BOLD] Answers  [BOLD] POS', '[BOLD] Answers  [BOLD] POS', '[BOLD] Answers  [BOLD] Pars', '[BOLD] Emails  [BOLD] POS', '[BOLD] Emails  [BOLD] POS', '[BOLD] Emails  [BOLD] Pars', '[BOLD] Newsgroups  [BOLD] POS', '[BOLD] Newsgroups  [BOLD] POS', '[BOLD] Newsgroups  [BOLD] Pars', '[BOLD] Reviews  [BOLD] POS', '[BOLD] Reviews  [BOLD] POS', '[BOLD] Reviews  [BOLD] Pars', '[BOLD] Weblogs  [BOLD] POS', '[BOLD] Weblogs  [BOLD] POS', '[BOLD] Weblogs  [BOLD] Pars', '[BOLD] WSJ  [BOLD] POS', '[BOLD] WSJ  [BOLD] POS', '[BOLD] WSJ  [BOLD] Pars']
Using Bayesian Optimization, we are able to outperform the baselines with almost all feature sets, except for a few cases (e.g., diversity and word embeddings similarity, topic and term distributions). Overall term distribution-based similarity emerges as the most powerful individual feature. Combining it with diversity does not prove as beneficial as in the sentiment analysis case, however, often yields the second-best results. Diversity features are stronger than for POS tagging and outperform the baselines for all except the Reviews domain. Similarly to POS tagging, term distribution-based similarity as well as its combination with diversity features yield the best results across most domains.