Which languages have you tried? My hunch is that to broaden the training data used for a LLM like OpenAI's gpt-3 or ChatGPT that they would inherently pull in content from several languages so it is baked into the model.