The Era of Experience

Why This Matters

David Silver and Richard S. Sutton's 2025 position paper is titled "Welcome to the Era of Experience". It predicts a shift from systems trained mainly on human-produced data toward agents that learn predominantly from streams of interaction.

The paper is a research agenda, not a theorem. Its historical sketch moves from simulation, through the current emphasis on human data, toward experience. Its operational proposal is a set of four characteristics, not a three-era taxonomy of "human knowledge, human data, and experience."

The Paper's Four Characteristics

Definition

Experience-based intelligence

Silver and Sutton expect leading systems to exhibit four linked traits:

1. Long streams: agents learn from streams of experience that grow with their lifetimes, rather than treating each interaction as an isolated query.
2. Grounded actions and observations: an agent's inputs and outputs are tied to consequences in an environment, including digital environments and interactions with people.
3. Grounded rewards: progress is evaluated from consequences in the stream of experience, rather than only from human judgments fixed in advance.
4. Planning and reasoning about experience: agents use learned world knowledge to anticipate outcomes and choose actions over longer horizons.

These are tendencies, not mutually exclusive bins. Human language, demonstrations, and feedback can remain part of an agent's experience when they arrive through grounded interaction. The paper argues for a change in the predominant source and structure of learning, not the removal of people from training.

Status of the General Thesis

Status: CANDIDATE. Experience-driven learning has produced striking results in bounded domains, but no general result proves that an experience-trained policy must outperform an imitation-trained policy. Such a comparison depends on the environment, reward specification, interaction budget, function class, optimization, safety constraints, and evaluation distribution.

The narrower claim retained on this page is directional: human-data-only training is likely to plateau, while continued interaction can generate new data and support learning beyond a fixed corpus. Silver and Sutton make a stronger forecast about superhuman agents, but the available evidence does not establish it in general. A model trained on human data can combine information and sometimes exceed individual human baselines without violating any statistical principle. A frozen model can also search or call tools at inference. If a system updates from the resulting grounded consequences over a long stream, that updating is experience in the paper's sense; tool use alone does not establish such learning.

FormalSLT status for the general thesis: NOT A THEOREM AS STATED. A formal comparison would first need a specified environment, policy classes, data and interaction budgets, learning algorithms, reward, and evaluation distribution. The proposition below is a separate supporting construction; it does not formalize Silver and Sutton's forecast.

What Experience Changes

Distribution shift

Behavioral cloning learns from a demonstrator's state distribution. At deployment, small policy errors can move the learner into states underrepresented in those demonstrations. DAgger and on-policy data collection reduce this occupancy mismatch by training on states the current policy visits.

They do not eliminate every form of shift. Environment dynamics, reward functions, users, sensors, and task definitions can still change. An agent can also learn unsafe behavior on its own trajectory distribution.

Proposition

An Absorbing-Error MDP Gives Quadratic Imitation Loss

Statement

Consider a horizon-$T$ decision process that starts in a safe state. The expert always takes the action that remains in the zero-loss safe state. While safe, an imitation policy independently takes a failure action with probability $\varepsilon$; that action enters an absorbing state with unit loss at the current and every later step. Let $\ell_t\in\{0,1\}$ be the loss at step $t$ and let $L_T=\sum_{t=1}^T\ell_t$. Then

$\mathbb E[L_T]=\sum_{t=1}^{T}\left[1-(1-\varepsilon)^t\right].$

If $0<\varepsilon\leq 1/T$, then

$$\frac{\varepsilon T(T+1)}{4} \leq \mathbb E[L_T] \leq \frac{\varepsilon T(T+1)}{2}.$$

Thus this construction turns error $\varepsilon$ on the expert's visited state into cumulative loss of order $\varepsilon T^2$.

Intuition

The expert visits only the safe state, so a small one-step imitation error looks harmless in the training distribution. At deployment, one error moves the learner into a state from which every later step is costly. The horizon amplifies the first mistake.

Proof Sketch

Loss is incurred at time $t$ exactly when at least one failure action occurred among the first $t$ steps. Independence while the process remains safe gives

$\Pr(\ell_t=1)=1-(1-\varepsilon)^t.$

Summing these probabilities proves the exact expectation. The union bound gives $1-(1-\varepsilon)^t\leq t\varepsilon$. The first two terms of the inclusion-exclusion lower bound give

$$1-(1-\varepsilon)^t \geq t\varepsilon-\binom{t}{2}\varepsilon^2 \geq \frac{t\varepsilon}{2},$$

because $(t-1)\varepsilon\leq1$. Summing the upper and lower bounds from $t=1$ to $T$ yields the claim.

FormalSLT status: OPEN. No corresponding theorem was found in the public FormalSLT theorem map during this review. The finite-state statement is a plausible formalization target, but the proof above is mathematical prose, not Lean evidence.

Why It Matters

This self-contained construction isolates the classical $O(T^2\varepsilon)$ occupancy-shift phenomenon discussed by Ross, Gordon, and Bagnell (2011). Collecting data on the learner's visited states can interrupt this failure pattern, which is the motivation for DAgger-style methods.

Failure Mode

This is an existence construction, not a claim that every imitation policy has quadratic loss. Recoverable errors, stable dynamics, closed-loop correction, and on-policy data can produce much smaller growth.

Cite as claim:era-of-experience::absorbing-error-imitation-loss

report a correction →

Adaptation over time

A model trained once on a fixed corpus does not absorb later events without another update. The same is true of a reinforcement-learning policy after training stops. Experience is a data source; continual adaptation additionally requires an update rule, memory, stability controls, and a way to avoid destructive forgetting.

Human feedback

RLHF is a hybrid. Policy-generated responses create new trajectories, while human comparisons or a learned preference model provide much of the reward signal. Human feedback can itself be grounded experience when it is part of an ongoing interaction. The Era of Experience thesis does not require an agent to invent its own objective, and doing so would create a separate alignment problem.

Evidence, Separated by Strength

AlphaZero: peer-reviewed, closed-domain precedent

AlphaZero learned chess, shogi, and Go from self-play under known game rules, without human game records. It showed that interaction plus search can surpass systems built with substantial human knowledge in domains with exact simulators, clear outcomes, cheap resets, and very large training budgets.

That is strong evidence for the mechanism in board games. It is not evidence that self-play supplies unlimited useful data in open environments, or that the same recipe transfers unchanged to medicine, robotics, or social interaction.

AlphaProof: peer-reviewed, hybrid mathematical system

The AlphaProof and AlphaGeometry 2 Nature paper reports a combined score of 28 out of 42 on the 2024 International Mathematical Olympiad problems, a silver-medal-equivalent result. AlphaProof solved three non-geometry problems. Its pipeline combined pretrained language models, supervised data, autoformalization at large scale, formal verification, and reinforcement learning.

This is evidence that verified feedback and search can extend a pretrained system. It is not a clean human-data-versus-experience experiment, and its multi-day computation is not comparable to a contestant's timed setting.

AlphaEvolve: white-paper evidence from evaluated search

AlphaEvolve combines language-model proposals, evolutionary search, and automated evaluators. Reported algorithmic improvements show the value of generating candidates and learning from machine-checkable outcomes.

The tasks and evaluators are supplied by people, so this is adjacent evidence for experience-centered search rather than an autonomous, open-ended learner. The public evidence is a developer-authored white paper, not the same evidence level as a peer-reviewed comparison.

SIMA 2: preprint evidence from a hybrid game agent

SIMA 2 studies a generalist agent acting in multiple three-dimensional game environments. The system combines a Gemini-based model, human demonstrations and instructions, and reinforcement learning from interaction. It is relevant evidence for grounded action and cross-environment transfer, but it remains a preprint and a hybrid training system with stated limits.

Related Arguments

The Bitter Lesson argues that general methods that scale with computation tend to beat systems dominated by hand-coded domain knowledge. The Era of Experience adds a claim about the source and temporal structure of data: interaction can keep producing consequences from which an agent learns.

Yann LeCun's "A Path Towards Autonomous Machine Intelligence" makes a related case for world models, planning, and sensory grounding. Agreement on those ingredients does not settle which architecture, training objective, or safety regime will work best.

Classical reinforcement-learning ideas are not discarded here. State, action, reward, return, policy evaluation, planning, partial observability, exploration, and off-policy correction remain central. The open question is how to make them work across long lifetimes and broad task distributions.

Open Questions

Environment fidelity. Board games have exact rules and cheap resets. Robotics and social interaction do not. A simulator can teach behavior that exploits simulation errors rather than solving the intended real-world task.

Reward grounding. Machine-checkable outcomes are powerful when the evaluator matches the real objective. Many important goals are delayed, multi-objective, socially contested, or only partially observable.

Sample efficiency. Experience can be expensive or dangerous. Model-based methods and offline data may reduce interaction costs, but the relevant scaling laws are not settled.

Continual learning. Long streams create nonstationarity, interference, memory demands, and safety questions. More experience is useful only if the learning system can retain and revise knowledge reliably.

Evaluation. A fair test must control compute, prior human data, interaction budgets, evaluator access, and deployment conditions. Comparing a hybrid system with a human-data baseline requires disclosing each source of training signal.

Common Confusions

Watch Out

The paper says language models are worthless

No. Silver and Sutton describe large language models as an important advance and a likely component of future agents. Their claim is that training predominantly on fixed human-produced corpora is unlikely to be the final learning regime.

Watch Out

Experience means no human involvement

No. People can supply initial data, instructions, feedback, environments, tools, and constraints. Human interaction can also be part of the experience stream. The central distinction is whether learning continues through grounded consequences rather than relying only on a static corpus.

Watch Out

On-policy learning removes distribution shift

It directly addresses the mismatch between demonstration states and the current policy's visited states. It does not make deployment stationary, guarantee safe exploration, or protect against a misspecified reward.

Summary

• The paper's operational framework has four characteristics: long streams, grounded actions and observations, grounded rewards, and planning or reasoning about experience.
• Its general prediction is CANDIDATE, not a proved performance inequality.
• AlphaZero is strong peer-reviewed evidence in closed games; AlphaProof is strong but hybrid evidence in formal mathematics.
• AlphaEvolve supplies developer-reported evidence for evaluated search; SIMA 2 supplies preprint evidence for a hybrid grounded agent.
• Experience reduces some imitation-learning limitations, but it does not automatically solve distribution shift, continual learning, reward specification, or safety.

Exercises

ExerciseCore

Problem

Audit an interactive assistant against the paper's four characteristics. It stores a user's history, calls tools, receives task-completion feedback, and plans several steps ahead. Which evidence would you need before saying that it learns from a long stream of experience?

Hint

Reveal Solution

ExerciseAdvanced

Problem

Is RLHF human-data learning or experience-based learning? Give the strongest case for each description.

Hint

Reveal Solution

ExerciseResearch

Problem

Design a comparison between behavioral cloning and experience-based learning in robotic manipulation. State the controls needed for a meaningful result.

Hint

Reveal Solution

References

Primary position papers:

• Silver & Sutton, "Welcome to the Era of Experience" (2025)
• Sutton, "The Bitter Lesson" (2019)
• LeCun, "A Path Towards Autonomous Machine Intelligence" (2022)

Systems evidence:

• Silver et al., "A General Reinforcement Learning Algorithm that Masters Chess, Shogi, and Go through Self-Play" (Science, 2018)
• Hubert et al., "Olympiad-Level Formal Mathematical Reasoning with Reinforcement Learning" (Nature 651, 2026; published online 2025)
• Novikov et al., "AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery" (white paper, 2025)
• SIMA Team et al., "SIMA 2: A Generalist Embodied Agent for Virtual Worlds" (preprint, 2025)

Imitation learning:

• Ross, Gordon & Bagnell, "A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning" (AISTATS, 2011)

Next Topics

• World Models and Planning: learning predictive models that support action selection