ChatGPT generates samples from a big but finite probability distribution which means it can be misled similarly to how image classifiers are misled by noise injected in images. A human that has an understanding of a concept will not be