Skip to main content
Back

Tencent's WorldCrafter Remembers Generated Worlds 47.6% Better on Revisit

A new video world model uses a camera-queryable 3D-aware memory to keep generated scenes consistent across long, revisiting camera paths.

Paper, Full Access
  • WorldCrafter is a video world model with an implicit 3D-aware memory queried by camera pose. Paper
  • Reports 47.6% relative improvement in long-horizon revisit consistency over the strongest baseline.
  • Memory encoder initialized from LagerNVS, jointly trained with the video DiT and readout module.
  • Max-coverage history retrieval picks complementary frames instead of the most similar ones.
  • Distilled WorldCrafter-fast streams at 16 fps on a 4-GPU machine using pyramid denoising.
  • Memory processing is 21.7x cheaper than depth-based spatial-memory pipelines. Project page

Interactive video world models generate a scene as a user moves a virtual camera through it. Their limited context windows create a persistent memory problem: after the earliest frames leave the window, revisiting a location can produce different walls, objects, or textures. WorldCrafter, developed by researchers at Peking University and Tencent’s ARC Lab, addresses that drift with an implicit 3D-aware memory queried by the requested camera pose.

The paper reports a 47.6% relative improvement in revisit consistency over its strongest baseline, better camera-control accuracy, and generation at 16 frames per second on four GPUs. The system joins work such as Google DeepMind’s Genie 3, Tencent’s HY-WorldPlay, NVIDIA’s Lyra 2.0, and Matrix-Game 3.5, all of which aim to turn video generation into controllable simulation.

Why generated worlds drift

Video world models predict future observations from user actions, usually by conditioning a diffusion transformer on recent frames and camera movement. A transformer can attend to a finite number of tokens, so long explorations eventually push early observations out of context. Existing memory designs manage that limit in three main ways:

  • Context memory: Retain earlier frames or cached attention states. Full-history attention raises compute and memory costs, while selective retrieval can omit useful views.
  • Spatial memory: Estimate scene depth, construct an explicit 3D representation, and warp it into the target view. Tight geometric alignment can restrict moving subjects and other scene changes.
  • Implicit memory: Compress history into learned tokens. Geometry-aware systems such as CineScene and GIM-World improve spatial recall, although geometry-estimation pretraining can preserve less appearance detail than reconstruction training.

WorldCrafter follows the implicit-memory approach and initializes its encoder from LagerNVS, a multi-view representation trained for novel-view reconstruction. That task requires the encoder to preserve scene geometry, color, texture, and object appearance across viewpoints.