Unsupervised Domain Clusters in Pretrained Language Models
Table 6: SacreBLEU scores for the data selection experiments. Highest scores per column are marked in bold.
['[EMPTY]', 'Medical', 'Law', 'Koran', 'IT', 'Subtitles', 'Average']
We can see that all selection methods performed much better in terms of BLEU than random selection. It is also nice to see that all selection methods performed better than using all the available data or the oracle-selected data when averaged across all domains, showing again that more data is not necessarily better in multi-domain scenarios and that data selection is a useful approach. Regarding a comparison of the data selection methods, Moore-Lewis performed better than Domain-Cosine, while Domain-Finetune performed best, showing the benefit of fine-tuning large pretrained models for the data selection task. Using the positively-labeled examples alone (Domain-Finetune-Positive) performed worse than using the top 500k examples but better than Domain-Cosine, while not requiring to determine the number of selected sentences.