Publikation
LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking
Lukas Henrik Helff; Quentin Delfosse; David Steinmann; Ruben Härle; Hikaru Shindo; Patrick Schramowski; Wolfgang Stammer; Kristian Kersting; Felix Friedrich
In: Computing Research Repository eprint Journal (CoRR), Vol. abs/2604.15149, Pages 1-8, arXiv, 2026.
Zusammenfassung
As reinforcement Learning with Verifiable Rewards (RLVR) has become the dom-
inant paradigm for scaling reasoning capabilities in LLMs, a new failure mode
emerges: LLMs gaming verifiers. We study this phenomenon on inductive rea-
soning tasks, where models must induce and output logical rules. We find that
RLVR-trained models systematically abandon rule induction. Instead of learning
generalizable patterns (e.g., “trains carrying red cars go east”), they enumerate
instance-level labels, producing outputs that pass verifiers without capturing the
relational patterns required by the task. We show that this behavior is not a failure
of understanding but a form of reward hacking: imperfect verifiers that check only
extensional correctness admit false positives. To detect such shortcuts, we introduce
Isomorphic Perturbation Testing (IPT), which evaluates a single model output under
both extensional and isomorphic verification, where the latter enforces invariance
under logically isomorphic tasks. While genuine rule induction remains invariant,
shortcut strategies fail. We find that shortcut behavior is specific to RLVR-trained
reasoning models (e.g., GPT-5, Olmo3) and absent in non-RLVR models (e.g., GPT-
4o, GPT-4.5, Ministral). Moreover, shortcut prevalence increases with task com-
plexity and inference-time compute. In controlled training experiments, extensional
verification directly induces shortcut strategies, while isomorphic verification elimi-
nates them. These results show that RLVR can incentivize reward hacking not only
through overt manipulation but also by exploiting what the verifier fails to enforce