NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 26·5 min read
Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL
TL;DR
One-step text-to-image generation faces challenges due to mismatches between reward optimization and generative dynamics, leading to reduced image quality. The Diff-Instruct with Diffused Reward (DIDR) framework was developed to align rewards more effectively across diffusion trajectories.
✦ Why It Matters
Engineers can leverage DIDR to improve the quality of text-to-image generation with fewer computational resources.
Key Takeaways
- Explore DIDR for efficient image generation tasks.
- Utilize DRS for better reward alignment in RL models.
- Implement DRP for quick estimation of reward metrics.
- Consider single-step generation to reduce computational costs.
Full Summary
One-step text-to-image generation has advanced significantly, allowing for real-time synthesis, but existing reinforcement learning (RL) methods struggle with reward optimization, often sacrificing image quality for higher rewards. To address this, Diff-Instruct with Diffused Reward (DIDR) was introduced, which utilizes a data-free trajectory-level alignment framework based on Integral KL minimization.
DIDR propagates an optimal reward-tilted clean-image distribution throughout the diffusion process, leading to a new metric called the Diffused Reward Score (DRS). Additionally, the Diffused Reward Proxy (DRP) was created as an efficient estimator for DRS using differentiable short-step denoising.
Extensive experiments demonstrated that DIDR consistently outperforms current one-step SDXL baselines, and when applied to a 6B DiT backbone, it surpassed a 50-step teacher model in preference alignment while requiring only one generation step. These findings suggest that DIDR can significantly enhance the efficiency and quality of image generation tasks.