Tencent's MixGRPO Cuts Diffusion Model AI Training Time by 71%
Tencent's MixGRPO cuts RLHF training time for flow-based image generators by up to 71% using a sliding-window ODE-SDE hybrid, accepted to ECCV 2026.
Takeaways
- MixGRPO is a new RLHF training framework from Tencent Hunyuan, accepted to ECCV 2026, that fine-tunes flow-based image generators using reinforcement learning.
- It introduces a sliding window mechanism that applies stochastic (SDE) sampling and gradient updates to only 4 denoising steps, using fast deterministic (ODE) sampling everywhere else.
- Compared to DanceGRPO, MixGRPO achieves ~50% lower training time; the MixGRPO-Flash variant cuts training time by 71% using higher-order ODE solvers outside the window.
- A pre-trained FLUX.1-dev checkpoint fine-tuned with HPSv2, ImageReward, and PickScore is publicly available on Hugging Face.
- Code and training scripts are open-source at Tencent-Hunyuan/MixGRPO; training requires 32 GPUs (4 nodes x 8).
- MixGRPO outperforms DanceGRPO, Flow-DPO, and Flow-GRPO on both in-domain and out-of-domain reward metrics across single and multi-reward settings.
Applying reinforcement learning to diffusion models is one of the hottest research directions in generative AI right now, but it comes with a brutal cost: you have to run the full denoising process, score the output, and backpropagate through every single step. Tencent's Hunyuan team just published MixGRPO, a training framework accepted to ECCV 2026 that cuts that cost dramatically, without sacrificing the quality gains that make RLHF worth doing in the first place.
The problem with GRPO on flow models
Flow matching models (think FLUX, SD3) generate images by learning a smooth transformation from noise to data, governed by an ODE. The trouble is that ODEs are deterministic: given the same starting noise, you always get the same image. Reinforcement learning, on the other hand, needs randomness to explore different outputs and compare them. The fix, pioneered by Flow-GRPO and DanceGRPO, is to convert the ODE into an equivalent SDE (stochastic differential equation) that adds controlled noise at each denoising step, enabling the model to generate a group of diverse outputs per prompt and rank them by reward.
GRPO (Group Relative Policy Optimization) is the same algorithm that powers reasoning in LLMs like DeepSeek-R1. It generates a batch of outputs, scores them with a reward model, and updates the policy to favor higher-scoring outputs relative to the group average, without needing a separate critic network. Methods like FlowGRPO and DanceGRPO still exhibit inefficiency due to the necessity of sampling and optimizing over all denoising steps specified by the Markov Decision Process (MDP). For a 28-step FLUX denoising process, that means 28 stochastic forward passes and 28 gradient updates per training sample, which is extremely expensive.