We are entering the world model era

Input camera

Input hand pose

Controls specify a candidate motion.

Cosmos 3 · generated future video

Generative models predict how the world changes under a chosen motion.

Worlds we can interact with

Genie 3 · action-conditioned exploration

The next view depends on what the user does.

Do we still need explicit 3D?

If the deliverable is pixels or actions, why prescribe a 3D representation?

Do we still need explicit 3D?

One reference image

Atlas · views generated along a camera path

A spatial experience can be generated without first delivering a complete 3D scene.

We can now simulate worlds in RGB

Atlas · generated worlds

Genie 3 · interactive exploration

Cosmos 3 · controlled futures

Compelling spatial experiences can emerge through RGB generation.

How can these predictions help an agent decide what to do?

What about reasoning?

How do these predictions help an agent choose what to do?

The same scene, now with questions about its structure.

Where are things?

Relative positions, distances, and occlusions.

What changes if I act?

How objects and observations change under an action.

Which motions are possible?

Reachability, clearance, and collision.

Does explicit 3D help us answer these questions?

The model can reason from context

The visual history contains evidence about the same world over time.

Spatial understanding can stay implicit in the model and its context.

As history grows: keep more context, compress it, or retrieve what matters.

3D as an efficient way to reason

Much of our reasoning is in service of acting in the physical world.

3D gives reasoning an explicit interface to the space where actions have consequences.

3D as an efficient way to reason

Reasoning over time needs memory.

3D provides a structural prior for persistence.

The same world need not be stored again at every timestep.

3D as an efficient way to reason

Stable structure, evolving objects.

LiveWorld · a persistent scene with dynamics that continue out of view

The world state can persist beyond the current context window.

3D as an efficient way to reason

Reuse spatial reasoning across time.

External example · NVIDIA cuRobo · depth-based mapping and motion generation

Keep the scene state

A 3D map carries geometry across observations.

Update what changes

New depth revises the current map.

Reuse the same queries

Check distance and collision for each new plan.

The time changes; the meaning of a spatial query stays the same.

3D as an efficient way to reason

A persistent memory layer for world prediction and action prediction.

Observations

New evidence

→

3D / object memory

Geometry

Features

Identity

Uncertainty

State · features · uncertainty

Persist · update · reuse

→

World prediction

Views and future states

Action prediction

Motion and constraints

Learn over persistent state, then reuse the model across time.

Next: spatial motion · spatial constraints · persistent memory

Two ways to use 3D for action

I’ll focus on PointAction and OmniGuide.

3D as an action interface

PointAction

Connect predicted 3D motion to robot control.

Video prediction → 3D point motion → robot actions

3D as a guidance field

OmniGuide

Use spatial constraints to guide a pretrained VLA.

VLA + 3D fields → guided actions

A spatial representation the model can predict and reason with.

Using world predictions to guide actions

Given an observation and an instruction, imagine a plan the robot can follow.

Make the world model’s plan explicit in 3D before translating it into action.

How does a video plan become an action?

PointAction · from imagined motion to robot control

A video model can predict a plausible manipulation.

The action decoder still has to infer…

Metric motion

How far should the gripper move in 3D?

Contact geometry

Where does the gripper meet the object?

Robot controls

Which commands realize that motion?

Grounding RGB in control requires robot-specific action supervision.

3D motion gives us a common interface

Make the predicted motion explicit before translating it into controls.

RGB · what the scene will look like

XYZ · where the geometry will move

Predict how the robot should move; learn how this robot executes it.

Learn motion broadly, adapt control locally

Separate learning what should move from learning a robot’s control space.

The bridge is 3D point motion: learnable from videos, usable for control.

Appearance and geometry, frame by frame

RGB carries appearance. The model predicts how the scene will look.

XYZ carries spatial motion. Pointmaps expose where robot geometry moves over time.

Geometry becomes a control interface. The action decoder reads masked robot points.

The action decoder reads the robot’s 3D motion

From robot videos to an adapted policy

First predict motion, then decode actions

$o$: RGB · $u$: XYZ + robot mask · $a$: actions · $s_t$: robot state · $l$: instruction

A shared representation with target-arm adaptation of the predictive model and decoder.

Adapting a video model to predict RGB and XYZ

Turning point trajectories into robot actions

Target-arm data supports RGB–XYZ post-training and decoder training; transfer is not zero-shot.

Training the action decoder

Simulation decoder: 6 Transformer blocks · hidden width 256 · 4 attention heads

Which geometry actually helps control?

RoboCasa365 sim, avg success (%), 100 rollouts per task.

In this setting, which spatial variables we expose changes the control result.

How good are the generated RGB and XYZ?

Bridge · RGB

Bridge · XYZ

DROID · RGB

DROID · XYZ

Generated RGB and corresponding dynamic XYZ pointmaps

How well does the policy generalize?

RoboCasa365 · success (%) · 100 rollouts per task

Spatial grounding helps control. Task generalization remains a separate challenge.

Adapting to two new robot arms

YAM and xArm7 — neither seen during 4D-video pretraining. Target-arm adaptation uses 20–50 expert trajectories per task.

xArm7 · pineapple

xArm7 · stack cubes

xArm7 · stack cups

YAM · stack cubes

YAM · pick pens

YAM · insert cups

(a) xArm7 — 100 rollouts / task

%	P&P	St.Cu	St.Cup	Avg.
GR00T N1.7	30.0	7.0	7.0	14.7
$\pi_{0.5}$	42.0	12.0	14.0	22.7
Ours	67.0	28.0	34.0	43.0

(b) YAM — 20 evals / task

%	St.Cu	PickPen	InsertCup
GR00T N1.5	0	20	15
$\pi_0$	0	10	15
Ours	20	60	50

xArm7: 43.0% vs. 22.7% average success; both new arms require target-arm adaptation.

Can 3D help without retraining the model?

3D can shape what a model learns and how a trained model acts.

Use 3D to check and refine an action

A VLA proposes actions; explicit spatial reasoning helps refine them.

VLA = Vision–Language–Action model · e.g., $\pi_{0.5}$ and GR00T N1.6

Repeat while denoising · policy weights stay fixed

3D makes the constraints explicit and directly computable.

OmniGuide: steering actions with 3D fields

Yunzhou Song, Long Le, Yong-Hyun Park, Jie Wang, Junyao Shi,
Lingjie Liu, Jiatao Gu, Eric Eaton, Dinesh Jayaraman, Kostas Daniilidis

Define an energy in 3D

$\mathcal L_y(X)$ scores a candidate Cartesian trajectory.

Turn it into a field

$F_y(X)=-\nabla_X\mathcal L_y(X)$

Attract toward a goal.
Repel away from obstacles.

The field lives in Cartesian space and guides generation in action space.

Robot time and denoising time

OmniGuide convention: $\tau=0$ noise, $\tau=1$ clean · $o$ collects the current policy context

Where do the 3D constraints come from?

All three score a clean Cartesian trajectory $X$.

Collision point clouds exclude the robot body and task-relevant objects. Human matching progresses forward in the demonstration.

Using a 3D field to guide denoising

1. Estimate a clean plan.
$\widetilde A^\tau=A^\tau+(1-\tau)v_\theta(A^\tau,o)$

2. Evaluate its Cartesian trajectory.
Decode actions, apply dynamics / kinematics, and compute $\mathcal L_y(X)$.

3. Change the sampling velocity.
$v_{\rm guided}=v_\theta-\lambda\,\mathrm{clip}(\nabla_{A^\tau}\mathcal L_y)$

The gradient changes the action sample; policy weights stay fixed.

Backpropagating a spatial correction to the action

Forward: evaluate the plan

Action sample → clean estimate → Cartesian poses → energy.

Backward: follow the blue arrows

Pass the spatial gradient through the robot mapping and the policy’s input.

Differentiate the action sample. The pretrained weights stay fixed.

Use the same 3D cost to select a starting seed

Before full denoising, compare a few short previews of possible plans.

Next: compare selection, refinement, and both.

Selection and refinement work together

Compare seed selection, denoising refinement, and their combination.

Refinement gives the larger gains; combining both performs best in this comparison.

RoboCasa + GR00T N1.6-3B · 50 trials per setting · ground-truth depth and object poses

Reaching the goal while avoiding obstacles

Official combined-guidance demo · RoboCasa
Semantic target + cluttered workspace

Combined guidance gives the best task success here. Collision-only guidance retains slightly higher safety.

One 3D trajectory, with both spatial objectives guiding its generation.

Improving a policy without retraining it

Real Franka robot · human-trajectory guidance
“open the top left drawer”

Aggregate metric	Base	Guided
Success	24.2%	92.4%
Collision avoidance	7.0%	93.5%

Published cross-setting aggregates, not the success rate of this one video.

Cost: roughly 2× the inference time. Reported control frequency: ~30 Hz to ~15 Hz.

Local spatial objectives still depend on perception, calibration, and a useful policy prior.

FUTURE DIRECTIONS

What’s the future of 3D and world models?

An agent that anticipates change, compares action consequences, and learns from new evidence.

Can spatial reasoning become a persistent part of the model?

World models beyond video generation

An agent’s ability to predict how the world changes and what its actions will cause.

Actions condition predictions; predictions guide actions.

FUTURE DIRECTIONS

Three questions remain open

Where can 3D make these choices more effective or efficient?

FUTURE DIRECTIONS · 1. STATE / MEMORY

Use 3D knowledge to build a latent world state

Reuse existing 3D knowledge

Encode observations into compact, spatially meaningful latents.

Carry state into prediction

Preserve object identity, geometry, relations, and motion.

Then learn what should persist, what should change, and what remains uncertain.

The geometry visualizes the representation. The state itself can stay latent.

FUTURE DIRECTIONS · 1. STATE / MEMORY

Maintain a state as the world changes

Keep stable structure, represent uncertainty, and revise the state when new evidence arrives.

Research questions: memory growth · update cost · consistency over long interactions

FUTURE DIRECTIONS · 2. LEARNING TO PREDICT

How do we learn changes in this state?

Here $z_t$ summarizes the available history; this is a proposed modeling interface.

Next: what the STARFlow family offers as generative foundations.

FUTURE DIRECTIONS · GENERATIVE FOUNDATIONS

STARFlow: sampling with a tractable density

A foundation for learning distributions over compact representations.

Scalable capacity. One deep Transformer block carries most capacity + a few efficient shallow blocks.

Compact modeling space. Model compressed autoencoder features to scale image generation to high resolution.

Sampling and density evaluation. An invertible flow supports sampling and exact density in its noisy latent space. The full image model uses an ELBO.

FUTURE DIRECTIONS · GENERATIVE FOUNDATIONS

STARFlow-V: extend the model through time

STARFlow-V · generation examples

Temporal factorization. A deep autoregressive flow carries temporal context; shallow flows model each frame.

Likelihood-based learning. The flow models compressed video latents. A separate causal denoiser refines visual detail.

Next: state transitions. Extend the framework with actions and spatial state, beyond visual continuation.

ONGOING

World-model planning with STARFlow

Use an action-conditioned rollout model to compare possible futures.

Preliminary rollout example · ground truth / sampled prediction / score-denoised prediction

Video: open-loop prediction under dataset actions. The loop above describes the planning direction.

FUTURE DIRECTION · CONNECTING MEMORY AND PREDICTION

Explicit memory, implicit world model

Read a persistent spatial state, predict action consequences, then update it with new evidence.

An architecture to explore: learn the read / update interface and test long-term consistency, control, and cost.

One last example

Roberto Nickson · September 9, 2026

Appendix: clean and noisy histories

PointAction convention: $\tau=0$ clean, $\tau=1$ noise · frozen VAE and text encoder, LoRA adaptation