I think the more relevant related concern is the (accidental-ish) plagiarism that happens if I tell chatgpt to write three paragraphs that intro SGD and the result approaches a copy/paste of training data.