Using J-K-fold Cross Validation to Reduce Variance When Tuning NLP Models
Figure 5: Performance of different J-K-fold tuning procedures for a logistic regression part-of-speech tagger over 1,000 different partitions (3 s.f).
['SD of chosen', 'SD of chosen', 'SD of the accuracy estimate']
Our most unstable tuning is by 1-10-fold CV. However when checked on the held-out population, the tuned model with the worst true performance only loses 0.05% accuracy from the parameter choice with best global performance. So although increasing J does reduce variability, it is not always necessary for effective tuning, as vanilla 10-fold CV produced consistently near-optimal models. The size of variability, however, is still a concern for reproducibility, as different random partitions can lead to choosing C as 10, 50 or 100. Once again, at each computational cost, the most effective tuning (in terms of gamma and variability of the performance estimate of the tuned model) is from the lower choice of K and higher J. Note that for C variability, there is an slight increase when moving from 1-10-fold to 2-5-fold CV, however, this corresponds to a large drop in the other variabilities and so 2-5-fold CV still provides more effective tuning overall. Also note that we see a larger reduction in gamma variability than C variability for larger J choices. Note that effective parameter tuning corresponds to a single predominately large bin, i.e. selecting a single parameter choice irrespective of the partitioning. Also note that the accuracy estimate for the model tuned by 1-10-fold CV is not stable enough to allow comparison with other models of close performance. To be able to reliably distinguish between our tuned model and alternatives with performance differing by only a couple of percentage points, we require higher choices of J.