For the robots.txt, make sure to also remove the wildcard character "*" (if any) and normalize double forward slashes + remove any comments (starting with " ") Aside from that, I think you could just feed this to ChatGPT and you got a worki