Asking ChatGPT what it'll do to avoid nuking a city is not dissimilar to asking a doorbell to see if anyone is home. The doorbell can do that, but it isn't aware of anything that's involved. It doesn't know why it works. It doesn't know anything.