Skip to main content
Back

PianoFlow Generates Realistic Pianist Hand Motion 9x Faster in Real Time

A flow-matching model uses MIDI as a training-only crutch to generate coordinated two-handed piano performances from audio, running 9x faster than prior work.

Paper, Full Access
  • PianoFlow generates two-handed piano motion from audio using flow matching, 9x faster than prior diffusion baselines.
  • Uses MIDI as a privileged modality only during training, distilled into an audio-only inference path.
  • Asymmetric role-gated attention explicitly models left and right hand as distinct, temporally coordinated roles.
  • Autoregressive flow continuation stitches chunks along the optimal transport path for real-time streaming of arbitrary length.
  • Evaluated on the PianoMotion10M benchmark of 116 hours and 10M hand poses.
  • Preprint available on arXiv; no public code or checkpoint released yet.

Generating realistic hand motion for a pianist from audio alone sounds niche until you notice how many downstream applications depend on it: virtual avatars, VR music instruction, animated performers in games, and robotic piano players. A new paper called PianoFlow tackles the task with a flow-matching framework that produces coordinated bimanual motion, streams in real time, and runs roughly nine times faster than the previous state of the art.

Where prior methods stumble

The reference dataset in this space, PianoMotion10M, was released as 116 hours of bird's-eye piano performance video with 10 million annotated hand poses. It shipped with a baseline that generates motion from audio through a position predictor and a position-guided gesture generator. Follow-up work like Separate to Collaborate pushed further with a dual-stream diffusion model that generates each hand independently, combined with a Hand-Coordinated Asymmetric Attention mechanism.

The PianoFlow authors argue these approaches share three weaknesses:

  • They lean on acoustic-only representations lacking symbolic priors, so the model never sees the note structure the audio encodes.
  • Their cross-hand interaction is inflexible, treating left and right with the same generic attention.