This ChatGPT example is really interesting. Notice how 1 in the second response is subtly wrongSGD uses small batches because the gradient on the full data is too noisy. Its a striking example of how the model can fail in an unusual way.