I think in practice most LLM based apps won't be like ChatGPT where it's a magic box. I think the training data is just a way to get the LLM 'ready' then the info will be in the prompt context. Like here's a rough interface I was tinkering