Should Embodied Intelligence
Care About 3D?
Talk @ EmbodiedAIinLife Workshop · Jun 3, 2026
"Should you care about 3D?" — domain by domain
An influential framing — domain by domain.
I basically agree. A sharp framing — and very reasonable.
The row this talk wants to look at more carefully is Robotics — given today's data picture.
Three works, one arc
3D as an interface — three roles for the same primitive.
3D as action
PointAction — dynamic point trajectories as a universal action interface. arXiv 2606.03943
PointAction — dynamic point trajectories as a universal action interface. arXiv 2606.03943
3D as guidance
OmniGuide — differentiable 3D energy fields steer any VLA at test-time. arXiv 2026
OmniGuide — differentiable 3D energy fields steer any VLA at test-time. arXiv 2026
3D as physical prior
PhysCtrl — point-trajectory physics-grounded video on 4 materials. NeurIPS 2025
PhysCtrl — point-trajectory physics-grounded video on 4 materials. NeurIPS 2025
Building on prior work — WVD (CVPR'25, Highlight): 3D as a supervision signal for video models.
PointAction — 3D points as a universal action representation
Embodiment-agnostic point trajectories as the interface between video prediction and robot control.
PointAction — Tong, Jiang, …, Gu et al., arXiv 2606.03943 · oriontmt.github.io/pointaction
Prior work — WVD: 3D as supervision, not output
XYZ images. Per-pixel global 3D coordinates — pixel-aligned, exactly the shape a video model already speaks. RGB and XYZ are stacked into 6-channel frames.
One DiT, three tasks. Joint RGB + XYZ training + flexible inpainting unifies single-image-to-3D, multi-view stereo, and camera-controlled video — without per-task heads.
The point: 3D enters as a training signal, not as a hand-crafted intermediate the policy must trust.
PointAction extends this from static images to dynamic point trajectories — and connects them to action.
WVD — Zhang et al., CVPR 2025 (Highlight)
The trick: pixel-aligned 3D coordinates
XYZ image. Per-pixel global 3D coordinate transformed into the camera frame. Same H×W as the RGB image — exactly the shape a video model already speaks.
Texture-free consistency. Two pixels with the same XYZ across views correspond to the same 3D point — direct supervision for multi-view geometry, no scene-specific tricks.
One DiT, 6-channel frames. Stack RGB + XYZ into a single diffusion target. No new architecture; the joint distribution is learnt by an off-the-shelf video DiT.
WVD — Zhang et al., CVPR 2025 (Highlight)
Stage 1: joint RGB + XYZ rollout
Width-wise modality fusion. RGB and XYZ pointmaps are independently encoded into the same VAE latent and concatenated along width — geometric tokens sit next to their visual counterparts inside DiT self-attention. (Inspired by 4DNeX.)
Frozen DiT + LoRA. Foundation video DiT stays frozen; a small LoRA adapter learns the joint RGB+XYZ rollout. Trained with flow matching under diffusion forcing on cross-arm robot videos (DROID + BridgeData V2).
Output: a 4D scene. Predicted RGB frames $\tilde o_{1:H}$ + pixel-aligned XYZ pointmaps $\tilde u_{1:H}$ — temporally and geometrically consistent. A separate video segmentation step extracts robot-centric 3D points for the decoder.
PointAction, arXiv 2606.03943
Stage 2: point trajectories → low-level actions
FPS sampling. Robot-centric pointmap → farthest-point sampling per frame.
PointNet-style encoder. Sampled points → token-aligned point features.
DiT denoiser. Conditional DiT denoises the entire action chunk in parallel; AdaLN injects diffusion step + initial robot state.
A small amount of paired robot data is only needed to train this decoder — the world model is unchanged across embodiments.
PointAction, arXiv 2606.03943
OmniGuide — 3D guidance fields for any VLA
Test-time scaffolding. Steer any diffusion-based VLA with task-specific 3D attractors and repellers — no extra robot data, no retraining.
OmniGuide — Song, Le, …, Gu, Eaton, Jayaraman, Daniilidis, arXiv 2026 · omniguide.github.io
How the gradient gets in
Velocity + score. Any diffusion / flow-matching VLA already produces a velocity $\mathbf{v}_\theta$. Bayes ties it to a posterior score: $\nabla_{\!\mathbf{A}^\tau}\log p(\mathbf{A}^\tau\mid\mathbf{y}) = \nabla_{\!\mathbf{A}^\tau}\log p(\mathbf{A}^\tau) + \nabla_{\!\mathbf{A}^\tau}\log p(\mathbf{y}\mid\mathbf{A}^\tau)$.
External gradient injects $\mathcal{L}_y$. Decode noisy actions to Cartesian poses $\mathbf{X}$, evaluate $\mathcal{L}_y(\mathbf{X})$, backprop through the robot kinematics — guidance vector lands back on the noisy action chunk.
Effect. The external field narrows the VLA's multimodal posterior toward task-effective + safe + physically-grounded actions.
OmniGuide, arXiv 2026 — Method overview, Fig 2
How PhysCtrl works — point cloud → physics → video
Point cloud lifting. Single image → segment → multi-view novel-view generation → 2048-point cloud reconstruction.
Physics-grounded trajectory model. Diffusion model conditioned on material, force, drag points; trained on 550K MPM + rigid-body simulations across 4 material families.
Track maps drive video. Projected 2D trajectories conditioning a pretrained video generator — physics guides pixels, not the other way around.
PhysCtrl — Wang et al., NeurIPS 2025
Four materials — elastic, sand, plasticine, rigid
Numbers. Trajectory generation: vIoU 77.6% · Chamfer 0.0028 (Motion2VecSets 24.9%, MDM 53.8%). Spatiotemporal attention + physics loss is the difference.
Ablation. Removing the physics loss collapses vIoU to 33.8%. The physics supervision is load-bearing — not a regulariser, the contribution.
PhysCtrl — Wang et al., NeurIPS 2025 · cwchenwang.github.io/physctrl
Thank you
Should embodied intelligence care about 3D? — maybe — at least as an interface, for now.
WVD (CVPR'25) · PointAction (arXiv 2606.03943) · OmniGuide (arXiv'26) · PhysCtrl (NeurIPS'25)
scan for more
jiataogu.me
Jiatao Gu · GMLR · Penn