Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense
Lin Long
Abstract
HERO, a reinforcement learning framework, combines verifier signals with reward-model scores to enhance reasoning in large language models, outperforming both RM-only and verifier-only methods.
Post-training for reasoning of large language models (LLMs) increasingly relies on verifiable rewards: deterministic checkers that provide 0-1 correctness signals. While reliable, such binary feedback is brittle--many tasks admit partially correct or alternative answers that verifiers under-credit, and the resulting all-or-nothing supervision limits learning. Reward models offer richer, continuous feedback, which can serve as a complementary supervisory signal to verifiers. We introduce HERO (Hybrid Ensemble Reward Optimization), a reinforcement learning framework that integrates verifier signals with reward-model scores in a structured way. HERO employs stratified normalization to bound reward-model scores within verifier-defined groups, preserving correctness while refining quality distinctions, and variance-aware weighting to emphasize challenging prompts where dense signals matter most. Across diverse mathematical reasoning benchmarks, HERO consistently outperforms RM-only and verifier-only baselines, with strong gains on both verifiable and hard-to-verify tasks. Our results show that hybrid reward design retains the stability of verifiers while leveraging the nuance of reward models to advance reasoning.
Community
HERO (Hybrid Ensemble Reward Optimization), a reinforcement learning framework that integrates verifier signals with reward-model scores in a structured way.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment (2025)
- A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models (2025)
- No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping (2025)
- $\lambda$-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences (2025)
- ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs (2025)
- Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training (2025)
- Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Thank you for the fantastic work on the HERO paper! The reward design is very insightful.
I have a quick question about the variance-aware reweighting. My understanding is that GRPO normalize advantages within each prompt's response group, e.g., Adv = (r - mean(r)) / std(r). If this is the case, the reweighting factor W would apply to both the numerator and the denominator (std(W*r) = W*std(r)), causing it to be canceled out in Adv.
So I suspect that the GRPO advantage normalization here is instead performed at the batch level rather than at the group level? This would preserve W in Adv. Could you please confirm if this understanding is correct?
Thanks
This is the author. Thanks for the reminder! We remove the advantage normalization during training. Sorry for forgetting mentioning it in the paper. Thanks for the reminder, will add it later.
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper