Zhejiang's DMT Turns Any Video Into Flicker-Free Sketch Animation on 3GB
A new method turns any video into a clean, jitter-free vector sketch animation by modeling stroke motion as smooth polynomial trajectories.
Takeaways
- New method converts videos to jitter-free vector sketch animations using polynomial control-point trajectories.
- Bernstein basis polynomials give uniform gradient sensitivity, converging in 500 iterations versus 1,000 with power basis.
- Sparse CoTracker points replace neural deformation fields, enabling stable results on 400-plus frame videos.
- Uses only 3.37 GB GPU memory for 400 frames, an 85% reduction versus SketchVideo at 50 frames.
- Continuous time representation enables free frame interpolation, turning 6 FPS input into 24 FPS output.
- Works cross-domain: 3D Unity animations and CogVideoX text-to-video both feed the same pipeline.
Turning a video into a hand-drawn sketch animation sounds like a solved problem in the age of diffusion models, but anyone who has tried it knows the result usually flickers like a broken neon sign. Strokes pop in and out, drift across the wrong body parts, and refuse to stay attached to the object they are supposed to describe. A new paper from Zhejiang University of Technology proposes a clean fix: stop treating each frame as an independent optimization problem, and start modeling stroke motion as a continuous polynomial in time. They call it Differentiable Motion Trajectories (DMT), and it produces vector sketch animations that stay coherent across hundreds of frames while using less GPU memory than every method it competes with.
Why frame-by-frame sketching breaks
The dominant approach for image-to-sketch conversion is CLIPasso, which represents a sketch as a handful of Bezier curves and optimizes their control points so that a differentiable rasterizer produces something CLIP thinks looks like the input. It works beautifully on still images. Applied frame-by-frame to generate sketch animations, it flickers badly, because nothing ties the topology of strokes together across time.
Follow-up work tried to patch this with neural representations of video geometry. SketchVideo used Neural Layered Atlases, and Fang et al. used Content Deformation Fields, both nudging strokes toward frame-to-frame consistency. The authors identify three problems that persist:
- Temporal popping and jitter remain because soft consistency losses cannot strictly enforce continuous motion.
- Semantics stay local, since only nearby frames influence each stroke, wasting information from the rest of the video.