[
Table 5: Performance on CoLA and STS-B, under different pretraining corpus of CSSL.
['[EMPTY]', 'CoLA', 'STS-B']
Another interesting point to study is what corpora should be used for training CSSL. In CERT, by default, we use the training data (excluding labels) of the target task for CSSL. We compare with the following setting. We randomly sample x sentences from the training corpora of BERT, where x is the number of training sentences in the target task. On the CoLA task, better performance is achieved when using the training data of CoLA for CSSL. On the STS-B task, better performance is achieved when using the BERT pretraining corpus for CSSL. From this study, we do not reach a clear conclusion regarding which one is better. In practice, it might be useful to try both and find out which one is more effective.