Cursor and Cornell Fix the Silent Reward Problem Blocking Agentic AI Training
Cursor researchers show how to blend textual judge feedback into on-policy RL without blowing up entropy or regressing on coding pass-rate.
Takeaways
- Cursor team releases report on stabilizing RL with self-distillation in agentic coding settings.
- Targeted guidance uses per-turn textual judge feedback as a per-token KL signal against a feedback-conditioned expert.
- Only applying feedback on failing turns preserves baseline coding pass-rate; always-on feedback regresses it.
- Low distillation weight prevents entropy spikes, length blowups, and subsequent pass-rate collapse.
- Including thinking tokens in the KL avoids oscillatory behavior on style criteria during training.
- No code or models released; tested in multi-turn software engineering agent environment.
Researchers at Cursor, Cornell, and UC Berkeley have released a paper on a persistent problem in agentic reinforcement learning: a policy cannot learn from a reward when it never samples the requested behavior. Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning proposes targeted guidance, a self-distillation method that converts judge-generated feedback into a token-level training signal. Under the paper’s tuned settings, the method teaches new behavior while preserving the model’s reported coding performance.
Why rewards can go silent
Agentic reinforcement learning trains models on multi-step tasks such as editing code, calling tools, and reporting progress. On-policy algorithms learn exclusively from trajectories generated by the current policy. Group Relative Policy Optimization, or GRPO, compares rewards among sampled trajectories to determine which outputs should become more likely.
If every sampled response fails the same criterion, those comparisons provide no direction for improvement. The authors describe the requested behavior as outside the policy’s support, meaning the model produces it too rarely for training to discover. Their baseline GRPO run could not learn objectives such as using an unfamiliar tool-call format or writing status updates as poems.
A scalar score records whether a response passed. Structured judge feedback can also identify the missing delimiter, formatting rule, or stylistic constraint. Targeted guidance feeds that specific information back into training.