University of Pennsylvania · Fall 2026
World Models
This course introduces world models—learned representations and predictors of environment dynamics for perception, planning, and decision making. We study how they support control and reasoning across reinforcement learning, video and 3D, multimodal agents, and robotics.
What is next
Course staff
Meet the instructional team. Instructor office hours are Thursdays, 4:30–5:30 PM in AGH 423.
TA office hours: Please see Ed Discussion for schedules, locations, and online meeting links.
Instructor
Jiatao Gu
Teaching assistant
Mutian Tong
Teaching assistant
Yong-Hyun Park
Teaching assistant
Enxin Song
Teaching assistant
Xinyue Ai
Tentative Schedule
21 lectures, one special topic, one invited lecture, one final project presentation, and one final exam, plus four confirmed no-class dates and one tentative cancellation, following the Penn academic calendar. Assignment and project deadlines are collected under Coursework. Slides and readings appear here as the semester goes on.
Representation, prediction, and interaction are recurring themes, not sequential phases. Later application units—including 3D and robotics—bring these questions back together.
Download course calendar (.ics) ↓- Latent World ModelsL3–L10
- Generative & Video ModelsL11–L14
- Spatial & Physical ModelsL15–L17
- Robotics & AgentsL18–L20
- 01
World Models: An Overview
Observation, state, transition, memory, action, prediction, simulation, planning, and reasoning.
- 02
World Models: History, Foundations, and Probabilistic Formulation
Trajectory distributions, latent state, partial observability, and one-step versus rollout objectives.
- 03
Environments, Simulators, and Rollouts
Environment interfaces, known transition rules, reset and step, observations, actions, feedback, and trajectory collection.
- 04
State-Space Models
Transition and emission models, filtering, smoothing, observability, and belief-state inference.
- 05
Self-supervised Representation Learning I
What representations preserve, how probes test them, and how reconstruction, masked prediction, autoregressive prediction, and contrastive objectives shape features.
- 06
Self-supervised Representation Learning II
Joint-embedding prediction, target encoders, masked latent prediction, and action-conditioned extensions.
- 07
Latent-Variable and Adversarial Models
Held virtually via Zoom. VAEs, the ELBO, posterior collapse, GANs, and explicit versus implicit generation.
- —
Tentative — No class
Class is tentatively canceled due to urgent instructor travel. Confirmation will be posted on Canvas.
- 08
Latent World Models
Tentative format change: virtual via Zoom; the link will be posted on Canvas. Autoregressive prediction, history and action conditioning, training versus rollout, and the World Models (2018) system.
- 09
Planning and Control with World Models
Held virtually via Zoom. Planning from LGSSMs to learned latent dynamics; MPC, CEM, MPPI, gradient-based action optimization, and planning limitations.
- 10
Policy and Value Learning with World Models
L6–L9 revisit: representation learning, latent prediction, LeWM, RSSM, and CEM, followed by an introduction to policy and value learning with Dreamer.
- —
No class — Fall Term Break
University break.
- —
No class — COLM 2026
Instructor conference travel.
- —
No class — COLM 2026
Instructor conference travel.
- 11
Dreamer, Diffusion and Flow Matching
Dreamer policy and value learning from imagined experience, VAE-to-diffusion foundations, DDIM, and flow matching.
- 12
Video Diffusion Models
DDIM, flow matching, guidance, U-Net and DiT architectures, latent and pixel generation, and bidirectional text-to-video and image-to-video models.
- 13
Video World Models and Fast Sampling
Joint video generation, spatial-temporal attention, action-conditioned autoregressive video, teacher forcing and diffusion forcing, flow maps, BOOT, pi-Flow, consistency, and MeanFlow.
- 14
Fast Sampling and Interactive World Models
Flow maps, BOOT, pi-Flow, consistency and MeanFlow; distribution matching with DMD, Drifting, and FD-Loss; autoregressive video rollouts, Self Forcing, Causal Forcing, and Self-Forcing++.
- 15
Special Topic: Are Normalizing Flows Good Candidates for World Models?
Normalizing flows as generative models and candidates for world modeling.
- 16
3D World Models I: 3D and 4D Representations
Points, meshes, voxels, fields, radiance fields, Gaussian splats, persistent point identity, visibility, coordinate frames, and deformation.
- 17
3D World Models II: Learning, Prediction, and Generation
Learning geometry and tracking from images, action-conditioned 3D prediction, and native and video-based spatial generation.
- 18
Neural Physics and Learned Physical Dynamics
Learned simulators for particles, meshes, fluids, and deformable objects; graph networks, neural operators, physical priors, and long-horizon stability.
- 19
World Models for Robot Learning I: Simulation, Control, and Sim-to-Real
Classical and learned dynamics for model-based control, system identification, domain randomization, and sim-to-real transfer.
- 20
World Models for Robot Learning II: Vision-Language-Action and World-Action Models
Vision-language-action policies, latent actions, video pretraining, and world-action models that jointly predict futures and robot actions.
- 21
LLM as World Models
Language-model world representations, state tracking, executable simulations, reasoning, and feedback for digital agents.
- 22
Evaluating World Models: Utility, Robustness, and Open Problems
Utility, controllability, calibration, OOD behavior, intervention, drift, latency, and failure.
- 23
Invited Lecture: Randall
Guest lecture by Randall. Topic to be announced. Penn follows a Thursday schedule on Tuesday.
- —
No class — Thanksgiving Break
University break.
- 24
Final Project Presentations
Student project presentations, discussion, and course synthesis.
- 25
Final Exam
In-class final exam covering the core concepts developed throughout the course.
Lecture slides
Lecture 1World Models: An Overview
Resources
A growing collection of useful blogs, tutorials, seminars, and systems. New links will be added throughout the semester.
Updated throughout Fall 2026
Essays & perspectives
6Tutorials & collections
6Talks & seminars
6Systems & demos
8Coursework
All assignment and project dates are collected here, separate from the lecture schedule. Specifications and grading weights are posted before each release.
Assignment 1
- Out
- Sep 10
- Due
- Sep 24
Assignment 2
- Out
- Sep 30
- Due
- Oct 19
Assignment 3
- Out
- Nov 2
- Due
- Nov 18
Final project
Final project data are now available. See Canvas for instructions and updates.
Explore a world-modeling method in one application domain, making the modeled state, transition, action interface, and evaluation criteria explicit.
Physical systemsVideo & spatial worldsRobot learningLanguage & digital agents
Proposal
Question, domain, method, and evaluation plan.
Checkpoint
Working system, early evidence, and risks.
Final project presentations
Results, failure analysis, and discussion.
Report + code
Final submission; exact format to be announced.
Logistics
Everything marked “to be announced” is confirmed and posted here before classes begin.
Recommended background
CIS 5190, CIS 5200, or equivalent graduate-level machine learning experience.
Official course listing ↗Office hours & course contact
Instructor office hours: Thursdays, 4:30–5:30 PM, AGH 423. Course questions and temporary meeting links are posted on Canvas.
Grading, late work & AI use
To be announced
Academic integrity
Penn’s Code of Academic Integrity applies to all work in this course.
Read the code ↗Accessibility
Approved accommodations are supported through the Weingarten Center.
Disability Services ↗Is this a reinforcement learning course?
No. We teach the RL and control machinery needed to understand how learned world models support decisions, but the course also covers representation learning, generative modeling, video and 3D, robotics, language, and digital agents.
Is the schedule final?
Class dates and major milestones are set. Topic order, readings, and guest speakers may still change, and per-session materials are linked from the schedule as they are ready.
What will the final project look like?
You explore a world-modeling method in one application domain. Final project data are now available. See Canvas for project instructions and updates.