Researchers at the Berkeley AI Research (BAIR) lab have released a reinforcement learning algorithm that bypasses temporal difference (TD) learning entirely. The new method, called Transitive RL (TRL), splits long tasks into smaller segments to solve problems that previously required complex, error-prone value estimation over thousands of steps.
In this article
Why standard off-policy learning struggles
Reinforcement learning divides into two categories: on-policy and off-policy. On-policy methods like PPO and GRPO rely strictly on fresh data generated by the current policy. They discard old information immediately. This approach works well for scaling but limits flexibility.
Off-policy algorithms remove this restriction. They can train on historical data, human demonstrations, or internet sources. This capability is essential for robotics, healthcare, and dialogue systems where gathering new data is costly. However, scaling off-policy learning to complex, long-horizon tasks has remained difficult.
As of 2025, researchers have solid recipes for scaling on-policy RL. A scalable, general-purpose solution for off-policy RL has not yet emerged. The primary obstacle lies in how these algorithms estimate value.
The limitations of Temporal Difference learning
Most off-policy systems use temporal difference (TD) learning, which relies on the Bellman update rule. This method calculates the value of a state based on the immediate reward and the estimated value of the next state.
The flaw is error propagation. When the algorithm estimates the value of a future state incorrectly, that mistake carries backward to the current state. Over a long sequence of actions, these errors accumulate, degrading performance.
To counteract this, developers often mix TD learning with Monte Carlo (MC) returns. They might use $n$-step TD learning, where the algorithm uses actual observed rewards for the first $n$ steps before bootstrapping. This reduces the number of recursive steps and limits error accumulation.
This hybrid approach has drawbacks. It does not eliminate the error accumulation problem, merely reduces the number of recursive steps by a constant factor. Furthermore, increasing $n$ leads to high variance and suboptimal results. Practitioners must carefully tune $n$ for every specific task, which is impractical for long-horizon problems.
A divide and conquer approach
The new proposal introduces a third paradigm: divide and conquer. The algorithm splits a trajectory into two equal-length segments and combines their values to update the value of the full path. This strategy reduces the number of Bellman recursions logarithmically rather than linearly.
Theoretically, this avoids the need to choose a hyperparameter like $n$ and mitigates the variance issues seen in $n$-step TD learning. The concept has existed for decades, dating back to Kaelbling (1993), but previous attempts failed to scale in practice because finding the optimal intermediate subgoal was computationally prohibitive.
Implementing the solution
Recent work co-led with Aditya Oberai addresses this gap by focusing on goal-conditioned reinforcement learning. In this setting, the policy must learn to reach any target state from any starting state. This structure naturally supports a divide-and-conquer logic.
The method relies on the triangle inequality. If $w$ is an optimal intermediate point on the shortest path between state $s$ and goal $g$, the distance satisfies $d^*(s, g) \leq d^*(s, w) + d^*(w, g)$. This translates to a transitive Bellman update rule where the value of $V(s, g)$ is derived from $V(s, w)$ and $V(w, g)$.
The practical challenge is identifying the optimal subgoal $w$. In tabular environments, one could enumerate all states to find the best option, similar to the Floyd-Warshall algorithm. This is impossible in continuous environments with large state spaces.
The TRL algorithm solves this by restricting the search space. It only considers states that appear in the dataset and lie between $s$ and $g$ in a sampled trajectory. Instead of a hard maximum, it uses expectile regression to compute a soft argmax. This minimizes a specific loss function over tuples of states $(s_i, s_k, s_j)$ where $i \leq k \leq j$.
This approach offers two advantages. It avoids searching the entire state space. It also prevents value overestimation caused by the max operator. The result is a system that recursively splits trajectories to handle long horizons without arbitrary chunking.
Performance on complex benchmarks
Researchers evaluated TRL on OGBench, a benchmark for offline goal-conditioned RL. They tested the algorithm on the hardest versions of humanoidmaze and puzzle tasks using datasets of 1 billion samples. These tasks require executing combinatorially complex skills over up to 3,000 environment steps.
TRL achieved the best performance on most tasks compared to strong baselines, including standard TD, Monte Carlo, and quasimetric learning methods.
A key comparison showed TRL matching the best individually tuned TD-$n$ performance across different values of $n$. The new method achieved this without requiring the user to set $n$ at all. By recursively splitting trajectories, it naturally handles long horizons without arbitrary choices of trajectory chunk length.
What it means
For engineers building systems that rely on offline data, such as autonomous robots or complex planning agents, this work removes a major tuning burden. Previously, practitioners had to balance the trade-off between variance and bias by selecting a specific step count $n$. TRL automates this process, allowing models to scale to tasks with thousands of steps without manual intervention on the horizon length.

