ShanghaiTech's MOVI Drops 3D-Aware Objects Into Videos Without Flickering
A new framework lifts reference images into 3D meshes and renders them from four angles to keep inserted objects consistent as cameras move.
Takeaways
- MOVI inserts objects into videos using multi-view 3D priors instead of a single reference image, from ShanghaiTech
- Reference image is lifted into a 3D mesh via Hunyuan3D 2.0, then rendered from four angles
- Dual-path injection combines latent concatenation for identity with a cross-attention feature bank for view retrieval
- CLIP-based semantic consistency check downweights broken renders when 3D reconstruction fails
- Depth and contour heads plus optical-flow loss handle occlusion, boundaries, and temporal flickering
- Achieves 23.22 PSNR and 0.7603 Box IoU on DAVIS 2017, beating VACE and MagicMotion baselines
Dropping a new object into a video sounds like a simple copy-paste job, but anyone who has tried it knows the pain. The object flickers, its texture warps when the camera swings around, it clips through foreground trees, and its edges bleed into the background. A team from ShanghaiTech University has proposed MOVI, a framework that tackles these failures by giving the generator a 3D understanding of the object before insertion, rather than a single flat reference image.
The core idea is straightforward: lift the 2D reference into a textured 3D mesh, render it from multiple viewpoints, and let the diffusion model pull the right view on demand as the scene moves. The resulting pipeline holds identity under camera rotation, respects occlusions, and produces sharper object boundaries than previous state-of-the-art systems.

Why single-image references keep failing
Recent controllable insertion methods like VACE, MagicMotion, and LoRA-Edit added spatial controls such as bounding boxes, masks, and trajectories on top of video diffusion. That gives you coarse control over where the object goes, but the appearance signal is still a single image or text prompt. These methods typically rely on text or a single reference to define appearance, which provides only partial information, so they often suffer from identity drift, distorted geometry, and severe artifacts when unseen views are required.
A second problem is that these systems mostly focus on the foreground and ignore how it interacts with a moving background. Camera motion and scene changes make it hard to maintain correct occlusion and temporal consistency, leading to incorrect layering, boundary artifacts, and flickering. Insert a bus into a street scene with foreground trees, and older methods either paint the vehicle on top of the branches or delete the branches entirely.