Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning

CHERRL (Controllable Hacking Environment for Rubric-based RL) is a controllable experimental testbed for studying reward hacking when an LLM-as-a-Judge (LaaJ) scores outputs against rubrics.

Rubric-based RL extends reinforcement learning beyond verifiable tasks, but policy models can exploit latent judge biases (e.g., verbosity or self-praise), leading to ineffective or unsafe training. In real deployments, such hacking is often subtle and entangled with multiple biases, making it hard to analyze, detect, and mitigate.

CHERRL addresses this by injecting known biases into LaaJ:

1. Controllable reproduction: Dual-judge rewards separate a clean reward from an isolated biased reward, so specific hacking behaviors can be induced and studied under controlled conditions.
2. Observable divergence: Tracking clean vs. biased rewards makes reward hacking and its onset explicitly measurable.
3. Analysis & detection: The environment supports analyzing bias discoverability/exploitability and developing agents that detect hacking onset from training logs.

Resources: The code and environment are publicly available at github.com/THUAIS-Lab/CHERRL.

Recommended citation: Wang, Xuekang, Zhuoyuan Hao, Shuo Hou, Hao Peng, Juanzi Li, and Xiaozhi Wang. (2026). "Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning." arXiv preprint arXiv:2606.04923.
Download Paper | Download Bibtex