Adapt across views
Residual cross-view attention, view embeddings, randomized view order, and masked replication support heterogeneous camera inputs.
VIDEO PRIORS. VARIABLE VIEWS. ROBOT ACTIONS.
X-World
Transferring Video Priors
to Variable-View Robot Control
Simulation benchmark
98.5%LIBERO mean success
Spatial · Object · Goal · LongZero-shot generalization
76.5%LIBERO-Plus success
Seven distribution shifts · No further adaptationReal-world manipulation
81.7%Six bimanual tasks
98 / 120 successful trials · 20 trials per taskRobot learning / Video-to-action transfer
X-World
Transferring Video Priors
to Variable-View Robot Control
1 School of Data Science, Fudan University2 Shanghai Innovation Institute3 Logos Intelligence Lab
† Corresponding author: lizhangfd@fudan.edu.cn
01 / The problem & our approach
From watching the world
to acting in it.
Video diffusion models capture object motion, contact, and task progression across diverse scenes. Transferring these representations to robot control requires handling different camera configurations, heterogeneous action spaces, and continuous execution.
We present X-World, a framework that connects a pretrained video backbone to a shared action expert through a fixed-length token sequence. A residual cross-view branch supports one, two, or three views while preserving pretrained per-view self-attention. A layer-aligned Q-Former compresses the visual representation, and dataset-specific projections retain each robot’s native state and action interface.
Training separates video adaptation from action-only pretraining, followed by joint target-domain post-training. At control time, a single backbone pass extracts visual features, while asynchronous action chunking supports execution. We evaluate the transfer on LIBERO, seven LIBERO-Plus distribution shifts, and six real-world bimanual tasks.
02 / Variable views, a shared interface
Keep the video prior.
Connect it to control.
Camera processing and action decoding meet through a fixed-length token sequence, allowing the two interfaces to adapt separately.
Compress the visual context
A layer-aligned Q-Former aggregates backbone features into fixed-length tokens that condition the action expert.
Act in native coordinates
Dataset-specific state and action projections connect a shared transformer to each robot’s native interface.
STAGE 01
Video adaptation
Learn variable-view video representations from robot and human videos.STAGE 02
Action pretraining
Freeze the video backbone and learn the Q-Former and action expert from action-labeled trajectories.STAGE 03
Target-domain post-training
Jointly update the model on LIBERO or real-world demonstrations with video and action supervision.03 / Evaluation
From benchmarks
to the real world.
Benchmark performance, distribution shifts, and real-robot trials reveal complementary aspects of the transferred representation.
81.7%
Mean real-world success
X-World succeeds in 98 of 120 trials across six tasks. Each policy uses the same 20 predefined initial placements per task; X-World achieves the highest reported success rate on every task.
98.5%
Mean LIBERO success
Evaluated on Spatial, Object, Goal, and Long with 50 episodes for each of 10 tasks per suite. The reported comparison includes methods with different pretraining data and compute budgets.
| Method | Spatial | Object | Goal | Long | Average |
|---|
X-World’s largest gain over X-VLA is on Spatial; X-VLA remains higher on Long. The small average gap motivates the distribution-shift evaluation.
76.5%
Zero-shot LIBERO-Plus success
The LIBERO-trained checkpoint is evaluated without further adaptation. Gains are strongest for robot, language, lighting, and layout changes; camera, background, and noise shifts remain challenging.
| Method | Camera | Robot | Language | Lighting | Background | Noise | Layout | Total |
|---|
04 / Understanding the transfer
What makes the prior useful?
98.5% → 89.5%
Video adaptation matters
Removing Stage 1 lowers mean LIBERO success. The two pretraining stages interact: their effects are not simply additive.
98.5% → 83.1%
Cross-view fusion matters
Removing residual cross-view attention causes the largest drop among the reported component removals. On Fold Towel, success falls from 12/20 to 0/20.
Explore the full training and component ablations
| Setting | Spatial | Object | Goal | Long | Average |
|---|
With video adaptation, action pretraining increases mean success from 97.4% to 98.5%. Without video adaptation, it lowers success from 92.1% to 89.5%. The aggregate results show an interaction, but do not establish its cause.
Explore view availability and real-time execution
46 ms cached model response · 91 ms deployed refresh · 30 Hz queued action execution. These measure different parts of the system. The selected six-action prefix spans 200 ms at 30 Hz.
05 / Real-world setup
Two arms.
Three viewpoints.
Two UR5e arms with Robotiq 2F85 grippers perform coordinated manipulation. A RealSense D455 observes the scene; two D405 wrist cameras capture local interactions.
Each task provides 30 minutes of VR teleoperation demonstrations. Data recording and action execution run at 30 Hz.
Watch the six tasks06 / Real-robot rollouts
Six tasks. Coordinated actions.
From object transfers and stacking to pouring and table clearing. These selected recordings illustrate X-World’s behavior; the quantitative results come from the full evaluation trials.
07 / Visual representations
Learning across viewpoints.
Future-video predictions illustrate the adapted representation across one, two, and three views. During control, X-World uses hidden features from a single backbone pass rather than decoding a future video.
View the multi-view future predictions
08 / Citation
BibTeX
@misc{gu2026xworld,
title = {X-World: Transferring Video Priors to
Variable-View Robot Control},
author = {Gu, Chun and Wei, Xiaofei and Chen, Yiming
and Zhang, Juntong and Zhang, Li},
year = {2026},
note = {Preprint}
}