Yeah, this works for inference. I haven't seen numbers for the gpt3.5 that backs ChatGPT, but I think it's safe to assume its probably taking 4-8 of the A100 to run inference on, even with savings like this to make the weights small