Sony CSL's LiveBand Jams With Live Musicians in Real Time on One GPU
LiveBand generates real-time music accompaniments on consumer hardware by replacing teacher forcing with adversarial sequence-level training, eliminating error drift entirely
Takeaways
- LiveBand generates real-time music accompaniments from live audio input on a single consumer GPU, with no lookahead into the future.
- It eliminates teacher forcing entirely -- the generator never feeds its own past outputs back as input, removing the root cause of error drift in autoregressive models.
- A GAN discriminator evaluates entire sequences rather than individual frames, providing holistic musical coherence signals during training.
- On the Slakh2100 benchmark, LiveBand outperforms StreamMusicGen on all metrics (audio quality, beat alignment, mix adherence) even when generating 1 second ahead of the live input.
- Inference runs at 2.1x real-time on an RTX 3090 with torch.compile, well within the 93ms latency budget.
- The demo page is live; no code or weights released yet.
Teaching an AI to jam with a live musician is harder than it sounds. The model has to listen, understand what's being played, and respond musically -- all in real time, without ever peeking at what comes next. A new system from Sony CSL Paris called LiveBand takes a fresh approach to this problem and, for the first time, makes it work reliably on a single consumer GPU without the musical drift that plagued earlier attempts.
The problem with jamming in real time
Most AI music generation systems are offline: they see the whole song, or at least a large chunk of it, before producing output. Live accompaniment is a fundamentally different challenge. Real-time music generation has been demonstrated in unconditioned or weakly conditioned settings, and offline systems have shown strong results for accompaniment generation, but the specific problem of live, mix-conditioned accompaniment remains substantially more difficult.
The core difficulty is something called exposure bias -- a well-known failure mode in autoregressive models. Standard training uses teacher forcing: at each step, the model is fed the ground-truth previous output as context, making training easy and stable. But at inference time, the model must feed its own predictions back as context. If an early prediction is slightly wrong, that error becomes part of the next input context; subsequent predictions are then made from an off-distribution state, which increases the chance of further errors. Repeating this feedback loop causes drift and long-horizon error accumulation. In live music, a slightly misaligned beat can quickly cascade into complete musical asynchrony.
There is also a second, more subtle problem: real-time generation requires the model to produce audio slightly ahead of the live input, to give the hardware time to decode and play it back. Recent work based on causal transformers over discrete neural audio tokens suggests that achieving adherence between the generated accompaniment and the input may require some lookahead into the future mix. LiveBand's central claim is that this is not a fundamental limitation -- it is a training mismatch that can be fixed.

The fix: ditch teacher forcing, use a discriminator instead
LiveBand replaces next-step predictive supervision with adversarial sequence-level supervision, and eliminates teacher forcing entirely by driving each step only with causal mix conditioning and independent noise. Instead of predicting the next frame given the correct previous frame, the generator receives a fresh Gaussian noise vector at each step alongside the current mix context. It never sees its own past outputs as inputs -- temporal dependencies are carried entirely through the transformer's internal attention state.