Unsupervised Domain Clusters in Pretrained Language Models
Table 7: Precision (p) and recall (r) for data selection of 500k sentences with respect to the oracle selection.
['[EMPTY]', 'Moore-Lewis p', 'Moore-Lewis r', 'D-Cosine p', 'D-Cosine r', 'D-Finetune p', 'D-Finetune r']
We perform an analysis on the selected datasets, where we measure the precision and recall of sentence selection with respect to the oracle selection. As also reflected in the BLEU scores, the Domain-Finetune method resulted in the highest domain recall with a minimum of 97.5, while Moore-Lewis and Domain-Cosine scored 89.4 and 78.8 respectively. We find these results very appealing given that only 2000 in-domain sentences were used for selection for each domain out of 1.45 million sentences. Also note that we used DistilBERT in these experiments: we believe that using larger, non-distilled models may result in even better selection performance (although at the price of larger computational requirements).