My tinkering so far reveals that ChatGPT readily recognizes SMILES, always confidently decodes them, doesn't appear to be doing simple string matching, and has a ~50:50 average for simple cases. I'm still blown away that this is a thing.