Ant Group’s Robbyant Unveils LingBot-VA 2.0: A Causal Video-Action Model Built Natively for Physical AI

Robbyant, the embodied AI unit within Ant Group, has released LingBot-VA 2.0. This is the first foundation model built natively for physical manipulation rather than digital video generation. The team pretrains the entire stack to handle robot control, avoiding the common practice of fine-tuning existing video generators.

What is LingBot-VA 2.0

Most video-action models reuse components designed for digital content. One is a reconstruction-oriented VAE. The other is a bidirectional video-diffusion backbone with an attached action module.

This setup creates three limitations. Pixel-reconstruction latents preserve appearance but carry limited physical structure. Iterative denoising over video tokens is too slow for closed-loop control. Generic video objectives never teach how actions reshape the world.

A fourth mismatch is structural. Backbones use bidirectional attention, while control unfolds strictly forward in time. LingBot VA Version 1.0 finetuned that stack into a causal model. Version 2.0 pretrains a causal DiT natively.

Version 1: The Semantic Visual-Action Tokenizer

Stage one replaces the compression-only VAE. Following RepWAM, the tokenizer adds two objectives to reconstruction.

Semantic alignment pulls visual latents toward a frozen Perception Encoder teacher. A latent-action objective extracts compact transition variables between consecutive latents. An inverse dynamics model predicts each latent action. A forward dynamics model decodes it into a transport map plus residual.

World states and actions now share one latent space. Unlabeled web video therefore carries action-relevant supervision.

Version 2: A Causal DiT With a Sparse MoE Video Stream

On top of that space, version 2 pretrains a causal DiT. It keeps the Mixture-of-Transformers layout of version 1.0. A video expert and an action expert share one causal self-attention. Each owns a separate feed-forward pathway.

The two streams scale asymmetrically. The video expert replaces its dense FFN with a sparse MoE routed layer. That layer holds 128 routed SwiGLU experts, top-8 routing, one shared expert. Load balancing follows the auxiliary-loss-free Loss-Free Balancing strategy. The action expert keeps a dense FFN at hidden dimension 768.

The video backbone is roughly 13.0B parameters, about 1.9B active. With the action expert and MCP heads, training covers about 15.3B parameters. Roughly 2.5B activate per token at inference. Training uses a rectified-flow objective with a hybrid Muon plus AdamW optimizer.

Where the Training Signal Comes From

Beyond architecture, two objectives shape what the model learns.

Multi-chunk prediction (MCP) fixes myopic supervision. Teacher forcing supervises only the next chunk, so the model can cut loss by copying appearance. MCP attaches three lightweight modules predicting the next three chunks. In ablation it matched the baseline’s 45k-step accuracy in 20k steps, a 2.3x training speedup.

Meanwhile, five objectives are co-trained rather than staged: T2I, T2V, TI2VA, ICL, and human-robot co-training. Sampling follows a coarse-to-fine schedule, from appearance grounding to video-action control. Keeping every objective alive avoids forgetting the earlier priors.

Hierarchical Planning

Chunk-level control cannot sequence long-horizon goals. Above the policy therefore sits a VLM planner, LoRA-finetuned with a frozen vision tower. It emits structured JSON: done, instruction, generation_instruction, local_scene_description. It runs at about 2 Hz behind an asynchronous shared buffer. The policy reads it at each chunk boundary, so planner latency never blocks execution.

Foresight Reasoning

Even with a sparse backbone, deployment hits a serial bottleneck. If the robot waits, model latency becomes control latency.

Foresight Reasoning therefore runs prediction and execution as asynchronous streams. While the robot executes chunk a_t, the video expert imagines its outcome. The action expert decodes a_{t+1} from that.

Running ahead risks drift. So each returning observation is encoded into the true latent z_{t+1}, overwriting the imagined one. A forward-dynamics grounding loss trains the video expert for this role.

# Pseudocode for the asynchronous rollout (Sec. 2.3.7, Eq. 29).
# Not runnable: policy, executor and encode() are placeholders.

C = init_kv_cache(encode(obs_0)) # feedback-grounded cache C_t
a = policy.action_expert(C) # cold start: first action chunk a_0

while not done:

executor.start(a) # execution stream, non-blocking

C_tmp = C + [a] # prediction stream: C_t u {a_t}
z_hat = policy.video_expert(C_tmp) # forward dynamics -> imagined z_{t+1}
a_next = policy.action_expert(C_tmp + [z_hat])

obs = executor.wait() # real observation of a_t returns
C = overwrite(C_tmp, z_hat, encode(obs)) # re-ground: z_hat <- true z_{t+1}
a = a_next

Performance

Consequently, evaluation covers simulation and real hardware. On RoboTwin 2.0, every model trains on 2,500 clean plus 25,000 randomized demonstrations, across 50 tasks.

Distillation cuts the video sampler from 5 steps to 2, and the action sampler from 10 to 2. FP8 TensorRT engines, a paged/ragged KV cache with FlashInfer attention, and host-side overhead removal supply the rest.

# Reproduces Table 3 of the report exactly. Runnable as-is.
K = 32 # low-level control steps inside one generated chunk

stack = [(“BF16 PyTorch async rollout baseline”, 927),
(“+ Consistency distillation”, 466),
(“+ Low-precision compiled execution”, 369),
(“+ Long-horizon attention optimization”, 272),
(“+ Runtime overhead reduction”, 142)]

for name, ms in stack:

print(f”{name:40s} {ms:4d} ms {round(1000 / ms * K):4d} Hz”)

print(“end-to-end speedup:”, round(927 / 142, 1), “x”)

Version 1.0 vs Version 2.0

The tokenizer ablation isolates row one. Swapping the WAN2.2 VAE for the semantic tokenizer lifts a 1.3