Skip to main content
Back

ETH Zurich's SpaceControl Lets Rough 3D Sketches Guide AI Asset Generation Without Retraining

A training-free method injects 3D geometry directly into the latent space of pretrained generators, giving users a fidelity dial without any retraining.

  • SpaceControl adds spatial 3D control to pretrained generators like Trellis without any fine-tuning.
  • An SDEdit-style approach voxelizes and encodes user geometry, then partially noises and denoises the latent.
  • A single tau_0 parameter trades geometric faithfulness against visual realism, sweet spot around 4-6.
  • Chamfer Distance beats Spice-E, Spice-E-T and Coin3D across ShapeNet chairs, tables, and unseen Toys4K.
  • Superquadrics, defined by only 11 parameters, act as compact sketching primitives for interactive editing.
  • Code and an interactive editor are available at the project repo.

Text prompts are ambiguous and reference images are painful to manipulate, which makes controlling the exact shape of a generated 3D asset one of the more frustrating gaps in modern generative pipelines. A team from ETH Zurich, Stanford, Technion and NVIDIA is proposing a different route with SpaceControl, a training-free method that lets you feed rough 3D primitives or full meshes directly to a pretrained generator and get back a detailed, textured asset that actually follows your geometry.

The trick is that no fine-tuning is required. SpaceControl introduces explicit spatial control into modern generative frameworks for text- or image-conditioned 3D generation, such as Trellis or SAM 3D, by directly encoding user-specified geometry into the latent space as explicit guidance. That means the same base model that produces text-to-3D outputs can suddenly accept a stack of blocky primitives as a shape prior.

SpaceControl teaser showing 3D assets generated from superquadric primitives and edited chairs and sofas

Why controllability keeps breaking

Existing spatially controlled 3D generators split into two camps, and both have real costs. Training-based methods extend generative models by fine-tuning them to accept particular forms of geometric inputs, for example voxel conditioning in LION or primitive- and mesh-based inputs in Spice-E, delivering explicit spatial control but requiring additional training that often reduces generalization. Guidance-based methods such as Latent-NeRF operate at inference time but typically incur substantial optimization overhead, and hybrid approaches like Coin3D still need fine-tuning plus test-time optimization on radiance fields.

In practice this means you either retrain a model per object class (and lose generality), or wait through long per-sample optimization loops. Neither works well when an artist wants to nudge a chair's backrest and see the result immediately.