Theres now a Python library for RLHF called TRLX! (The same reinforcement learning strategy used in training ChatGPT) It works well with Hugging Face models, supports multiple RL strategies, and requires very little code!