Using J-K-fold Cross Validation to Reduce Variance When Tuning NLP Models
Table 1: Standard deviation (SD) performance of J-K-fold tuning on the IMDB dataset over 1,000 different partitions (3 s.f).
['[EMPTY]', 'SD of chosen', 'Range of chosen', 'SD of the accuracy']
We see that between 1-10-fold CV and 2-5-fold CV, and between 2-10-fold, 4-5-fold and 1-20-fold (choices of equivalent computational cost), we have the least variability when reducing K and using the saved computation for increasing J. Note that the naive approach of using all available computation to increase K (1-20-fold CV) produces a wider range of parameter choices than 2-10-fold and 4-5-fold CV. This is despite any bias reductions from choosing a higher K and so we do not recommend 1-J∗K-fold CV. This analysis questions the rule of thumb selection of K=10 common in NLP, as 2-5 CV seems to produce superior tuning at the same cost, a hypothesis we now test across a range of NLP tasks.