Reasoning got cheap: what the RL wave actually changed

For a while, the story we told about reasoning models was a story about scale: bigger models, more data, longer chains of thought. Then in January 2025 DeepSeek-R1 reframed it as a story about incentives — and the eighteen months since have been about making those incentives cheap.

The result that reset the field

DeepSeek-R1’s headline was almost provocative in its simplicity: you can incentivize advanced reasoning through pure reinforcement learning on a base model, with rule-based rewards (GRPO) and no supervised reasoning traces at all. The R1-Zero variant developed self-verification, reflection, and dynamic strategy-switching on its own — including a now-famous spontaneous “aha moment” mid-derivation — and matched supervised counterparts on math, coding, and STEM. It shipped open-weight and was later peer-reviewed in Nature.

The important part isn’t the benchmark numbers. It’s that reasoning stopped looking like an emergent property you buy with parameters and started looking like a behaviour you can train for directly.

Then the wave made it efficient

If R1 proved the recipe worked, the next year was about the recipe’s economics. Three results stand out.

• ProRL (NVIDIA) asked whether prolonged RL genuinely expands a model’s reasoning boundary or just re-weights what the base model already knew. With KL control, reference-policy resetting, and a diverse task suite, their models outperformed the base even on tasks where the base failed at any number of samples — evidence that RL was adding capability, not just sharpening it.
• s1 went the other direction on cost: fine-tune on just 1,000 curated reasoning traces and apply “budget forcing” — literally appending the word “Wait” to make the model double-check itself. The resulting s1-32B beat o1-preview on competition math by up to 27%. No proprietary pipeline required.
• DLER (NVIDIA) attacked the tax reasoning models pay in tokens. It showed the accuracy loss from length penalties comes from inadequate RL optimization, not the penalty itself — and with a better recipe cut output length by 70%+ while surpassing prior accuracy. One variant reduced response length ~80% on math with better results.

Put together: reasoning became something you can train (R1), extend (ProRL), reproduce on a shoestring (s1), and then compress so it doesn’t bankrupt your token budget (DLER).

Why I care about this from the agent side

Most of what I build are agents — systems that take actions on your behalf. The blocker for agents was never that models couldn’t reason; it’s that reasoning was slow and expensive enough that you couldn’t afford to do it on every step of a multi-step task. Cheap, controllable, length-aware reasoning changes the calculus: an agent can afford to actually think before it clicks a button, and you can afford to let it.

The reasoning-on/off toggles now shipping in open model families are the productized version of this. The research made deliberation a dial. The interesting design work is deciding when to turn it up.