Intro

Reinforcement Learning (RL) has long been deep learning’s eccentric twin, elegant in theory, messy in practice. Once a niche behind AlphaGo and RLHF, it’s now moving to the core of intelligent systems as AI shifts from prediction to action.

The key question has changed: can models keep learning after training stops?

If pretraining taught models to understand the world, RL teaches them to operate in it. From RLHF to test-time RL and meta-RL, the pattern is clear, progress now comes from feedback, not scale. Compute scaling powered deep learning. Feedback scaling will power the agent era. The next leap won’t come from more parameters, but from tighter loops, every action making the next smarter.

In labs like OpenAI, Anthropic, and xAI, RL is no longer fine-tuning. It’s the next scaling frontier, the foundation of self-improving systems.

This essay distills insights from closed-door roundtables with RL researchers and engineers across frontier labs and companies, capturing emerging consensus around one idea: RL scaling isn’t a training trick, it’s the next system paradigm.

RL Takes the Lead: The Golden Age of Post-Training

• RL is rapidly moving from afterthought to centerpiece. For most of the past decade, nearly all compute went into pretraining, which is static, one-way learning from vast datasets. RL was a finishing touch, a few percent of total spend. That balance is changing fast. Frontier labs like DeepMind, xAI and Cohere now devote almost equal budgets to RL and pretraining, treating it not as fine-tuning, but as the second scaling curve of AI. Across the industry, the ratio is shifting from roughly 70:30 to 50:50, a quiet reallocation of compute that signals a new frontier in model development.

• Enterprise RL is emerging as the new Palantir model. Instead of training once and deploying forever, companies are embedding RL directly inside real customer environments. Each workflow, from customer support to contract analysis, becomes a living feedback loop. The model doesn’t just serve; it learns. Over time, a simple embedding model for search or summarization evolves into a self-improving system that adapts across departments. The enterprise itself becomes the training environment.

• Agentic RL is driving the next wave of investment. Once models can act, invoking tools, planning steps, pursuing goals, they must also learn from the consequences of those actions. As customer domains multiply, RL shifts from a tuning method to the core learning engine behind multi-domain agent systems. Every decision and every correction becomes data, the raw material of compound intelligence.

• RL is the new source of differentiation. Pretraining sets the ceiling; RL defines the behavior. As model architectures converge, the competitive moat moves to how effectively teams design reward functions, craft feedback environments, and engineer dense learning signals. RL doesn’t just make models smarter, it makes them distinct. In a world of similar weights, behavior becomes the brand.

The Engineering Reality of RL Scaling

• RL scaling now plays out on ten-thousand-GPU battlefields.

Frontier labs are running RL training loops across tens of thousands of GPUs, alternating nonstop between inference and learning. On-policy setups react instantly but waste compute, nearly 85% of time goes to inference rather than training. Hybrid strategies, which blend on- and off-policy methods, make better use of heterogeneous hardware but at the cost of massive bandwidth and sync overhead.

At this scale, RL becomes less about clever algorithms and more about systems choreography, keeping thousands of moving parts aligned without missing a beat.

• System consistency remains an unsolved frontier. Tiny numerical differences between frameworks — SGLang, vLLM, PyTorch, FSDP — can compound into major drift across long rollouts. Teams face a fundamental trade-off:

1. Go slow and stay stable, ensuring reproducibility but limiting throughput.
2. Go fast and drift, gaining efficiency at the cost of determinism.

Every frontier infra team quietly picks a spot on that curve. In large-scale RL, stability isn’t a feature; it’s an economic choice.

• The real bottleneck isn’t GPU count, it’s orchestration. At frontier scale, RL systems start to look less like training pipelines and more like distributed operating systems. They route rewards, rollouts, and replays across clusters the way networks route packets. The next breakthroughs won’t come from more silicon, but from smarter scheduling, which systems that learn how to learn efficiently. This is where the next 10× efficiency curve will come from.

The Inversion of Data and Reward

• The hard part of RL isn’t data, it’s reward. In supervised learning, more data usually helps. In reinforcement learning, it can hurt. Massive datasets mean nothing without clear, measurable success signals. That’s why robotics and autonomous driving, despite oceans of data, still struggle to scale. Without a working definition of “what good looks like,” more data just makes models confused faster.

• Imitation learning ≠ reinforcement learning. Imitation learning copies demonstrations; reinforcement learning discovers strategies. Most robots and dialogue systems are still replaying what they’ve seen, not optimizing for outcomes. Replication creates competence; feedback creates understanding.

• Reward hacking is a feature, not a bug. When a model exploits loopholes in your reward function, it isn’t misbehaving, it’s auditing your objective. Each failure exposes what you failed to specify. Closing those gaps turns reward design into a form of meta-learning, a feedback loop where both humans and machines learn to define success more precisely.both humans and machines to specify better goals.

• Diversity drives generalization. The richest RL progress comes from domains with dense, verifiable feedback like coding, math, logic reasoning. These structured environments compress experience into reusable abstraction, pushing systems toward transferable intelligence. That’s why frontier labs now use structured reasoning as the backbone of their data streams, not to chase leaderboard gains, but to scale generalization itself.

Environment as the new Bottleneck

• The best RL environment is the real one. Simulations can approximate reality, but only production systems expose the true constraints. Real environments have what synthetic ones can’t: latency, side effects, and unexpected dependencies that make optimization meaningful. The ideal path forward isn’t safer sandboxes, it’s letting models learn inside live systems, where every action carries real consequences and authentic feedback. Only the real world bites back, and that’s what makes it valuable.

• Environment consistency is the biggest barrier to scaling. The real world is noisy, unpredictable, and constantly changing. In enterprise systems, think Salesforce, internal databases, or multi-API workflows, even a small schema change or network delay can break a reward chain. One dropped packet can derail credit assignment. Until the environment itself becomes stable, even perfect algorithms can’t scale cleanly.

• The only stable environment is the one you control. This is the Tesla principle: if every car on the road were a Tesla, full self-driving would be easy. The same logic applies to RL. A unified interface ecosystem dramatically improves reward signal quality and feedback consistency. Owning the environment isn’t just a technical edge, it’s a strategic one. Control the environment, and you control the feedback loop.

• The long tail defines RL’s scaling speed. In domains like autonomous driving, recommendations, and ad optimization, environment diversity dictates how fast systems generalize. The more structured and standardized feedback, the faster the learning curve. The real test lies in the long tail those rare, messy, edge-case interactions that separate research prototypes from production intelligence.

Searching for the Scaling Laws of RL

• Real-world RL still lacks a clear scaling law. Language models follow predictable curves: more compute, better performance. Physical systems don’t. In robotics and embodied AI, data is expensive, feedback is delayed, and friction is literal. Constraints like latency and wear make linear scaling nearly impossible. A new generation of “physical intelligence” companies is now probing this gap, trying to map how learning behaves when every experiment costs real time, energy, and risk.

• Test-time training is the next slope-changer. Ilya Sutskever once said he’s “a believer in test-time training” that models should keep learning while deployed. When a robot encounters a new obstacle, it shouldn’t wait for retraining; it should adapt in place. Test-time learning closes the gap between knowing and doing, turning deployment into a continuous feedback loop.

• Training–inference mismatch is driving adaptation research. Supervised fine-tuning enforces correctness; RL encourages exploration. Test-time adaptation fuses the injecting controlled uncertainty during inference so systems can reason, experiment, and recover on the fly. Instead of breaking the distribution shift, they learn through it.

• The reward loop is the new scaling loop. Pretraining scaled parameters; RL scales interactions. The next scaling law won’t chart FLOPs against performance, but feedback throughput against intelligence growth. How many high-quality feedback cycles can a system complete per day? That rate, not model size, will define the next era of progress. In the end, intelligence compounds through loops, not layers.

From RL to World Models

• At its core, RL is about maximizing the probability of high-reward outcomes. Every scaling law in reinforcement learning boils down to this: enabling models to navigate high-dimensional spaces more efficiently toward regions of high reward. Exploration is compute; compression is generalization. The more efficiently a model explores, the more effectively it learns to compress experience into reusable knowledge.

• RL is statistical learning with consequences. When each sample affects the next, learning becomes a closed feedback loop. Knowledge shapes experience, and experience reshapes knowledge. At sufficient scale, reward optimization becomes the new form of compression, a process of memory → abstraction → generalization. Rather than memorizing data, the system learns principles that consistently produce reward.

• From AlphaGo to agentic RL: closed to open worlds. AlphaGo mastered a world with perfect rules; today’s agents operate in worlds with fuzzy goals and shifting boundaries. The next breakthroughs will come from systems that can set their own objectives, run their own experiments, and evaluate their own success, entities that don’t just follow rewards, but create them. This is the evolution from reinforcement learning to world modeling, from optimizing outcomes to understanding the environment itself.

RL Meets the Real Economy

• RL is already embedded in the engines of the modern economy. From advertising to recommendation to logistics, reinforcement learning is quietly becoming the invisible infrastructure behind large-scale optimization. Meta, ByteDance, and other giants are rearchitecting their legacy recommender systems into LLM-based RL agents, models that automate ad targeting, adapt strategies on the fly, and close the loop through self-feedback. The commercial upside is enormous: these are trillion-dollar markets where every marginal improvement compounds into real revenue.

• Recommendation systems were the original RL playground. For years, click-through rate was a crude form of reward. Traditional recommendation pipelines have always mimicked reinforcement learning, observe, act, measure, repeat. What’s changing now is that RL scaling makes this learning end-to-end and continuous. Recommenders have ideal properties for RL: short horizons, dense signals, and real-time feedback. At massive traffic scale, like social platforms or marketplaces, true end-to-end RL systems are finally within reach.

• Dense and verifiable feedback is the gold standard. RL thrives where feedback is clear, measurable, and immediate, domains like code generation, search ranking, and ad optimization. These environments make success explicit: the code runs, the link gets clicked, the ad converts. That’s why these verticals are emerging as the first commercial proving grounds for large-scale RL.

• Reward streams are replacing data lakes. The old data flywheel was passive, collect, label, retrain. The new one is live and self-reinforcing: every user interaction becomes a reward signal. Instead of static datasets, we now have real-time reward loops, systems that don’t just learn from the past, but improve from the present.

The Future of RL Scaling

• RL scaling may rewrite the laws of AI scaling. If Chinchilla defined the last era of AI by plotting parameters against performance, the next will plot interactions against reward. Intelligence will no longer be determined by how much compute you spend, but by how many meaningful feedback loops your system can complete. The next scaling law is not about FLOPs, it’s about feedback frequency.

• RL scaling is system scaling. AI is shifting from model-centric to system-centric design. As one engineer put it, “The algorithm is the easy part, the hard part is the system.” The real frontier isn’t better policy gradients, but the orchestration of rollouts, traces, and evaluations into a closed feedback system that never stops learning.

• Reward density will define computational efficiency. Sparse rewards burn compute without progress; dense rewards compound like interest. The new efficiency metric isn’t FLOPs per token, but learning gain per unit reward. The richer the feedback, the steeper the learning curve, turning signal density into the new scaling accelerator.

• Counterfactual learning is the next frontier. Instead of only logging what went right, counterfactual RL tracks what could have happened. By storing complete state trajectories, models can replay alternative decisions, “what if I had taken another path?”, and extract additional learning signals from hypothetical experience. These virtual replays act like simulated experiments, dramatically improving data efficiency.

• Trace completeness and diversity are the new scaling variables. A large, diverse replay buffer is itself a counterfactual dataset. It lets models learn attribution and causality at an aggregate level, stabilizing long-horizon feedback. At massive rollout scale, traces begin to approximate human-like reasoning: learning not from single outcomes, but from distributions of possible worlds.

• The endgame is an end-to-end self-evolving system. When rollouts, traces, and evaluations fully automate into a continuous loop, RL ceases to be a training stage, it becomes the system’s lifecycle. At that point, an AI system doesn’t just run models; it runs evolution. This is the true promise of RL scaling: from static intelligence to living systems that grow smarter simply by existing.

• Memory, Weight, and Continuous Learning. Human memory alternates between absorbing and consolidating experience. For models, this suggests dual-memory architecture: one for dynamic retrieval, another for structural rewiring.

There appear to be two tiers of “memory.”

1. The first is persistent learning: it doesn’t alter model weights but improves how the model dynamically retrieves and uses external context.
2. The second is a deeper algorithmic update: it permanently rewires the model’s internal knowledge structure.

In the ideal setup, generalizable patterns live within the model, while factual or short-term memory is offloaded and accessed through real-time lookup. This layered memory design defines how a model can continuously learn without catastrophic forgetting.

• Automating AI Research: One of OpenAI’s long-term ambitions is to create an automated AI researcher, a system capable of conducting research autonomously. Unlike coding or math, AI research has a much longer reward cycle, as it requires evaluating accuracy and efficiency across extended time horizons. OpenAI’s MLE Bench, which spans 76 research tasks, is an early step in that direction, allowing models to explore research pathways on their own. This is a move toward meta-learning: systems that don’t just solve problems, but also design, verify, and improve the algorithms that solve them. This is the beginning of self-improving AI research, where discovery itself becomes a reinforcement loop.

• Open Questions

1. Can RL sustain improvement across all domains? If a model can simultaneously exhibit strong generator and discriminator capabilities, it could self-improve through ongoing comparison and optimization across diverse tasks.
2. Can synthetic data drive self-evolution? Imagine a system that generates and refines its own responses through synthetic data—what some call training-free GRQ. With a rich enough prompt space, a model could bootstrap its own improvement loop without human labels.
3. What happens if RL scaling detaches entirely from human supervision? We may enter an era where AI evolves through self-assessment, self-correction, and self-reinforcement—a fully closed learning loop that turns the act of “doing research” itself into an emergent property of the model.