Using J-K-fold Cross Validation to Reduce Variance When Tuning NLP Models
Figure 8: Performance of J-K-fold tuning when choosing the width of the LSTM (3 s.f).
['SD of chosen bias', 'SD of chosen input', 'SD of the accuracy estimate']
We perform two experiments: tuning the number of nodes in the LSTM layer (known as width) across the grid {10,20,30,..,90}, and separately tuning the amount of l2 regularisation on the inputs and biases (for a fixed width of 50) each across {0.00001,0.001,0.1}]. In contrast, tuning with 8-5-fold produces much more consistent choices, the majority of the time choosing 70 for the optimal width and 0.001 Although variability in our chosen LSTM parameters does reduce as we increase J, it remains significantly higher than the gaps in our parameter grid. When coupled with the relative stability of the accuracy estimates of this tuned model, this suggests that the performance differences between the most frequent choices by 8-5-fold CV are small. However, to consistently tune the model to this specificity we require larger choices of J than 8.