NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 25·12 min read
DrawVideo: Generating Long Video from Storyboard Keyframe Sketches
TL;DR
Generating long videos with AI typically relies on single text prompts, limiting control over visual composition and motion across extended sequences. DrawVideo introduces a sketch-guided storyboard framework that decomposes videos into independently controllable shots, each defined by sketches (pose/layout), appearance prompts (identity/style), and motion prompts (temporal dynamics).
✦ Why It Matters
Engineers can now generate long videos with fine-grained control over composition, motion, and visual consistency using sketch-based storyboards instead of unstructured text.
Key Takeaways
- Utilize DrawVideo for generating videos from storyboard sketches.
- Leverage the SketchLongVideo dataset for training models.
- Implement hierarchical strategies for video synthesis.
- Focus on user control over video elements for better outcomes.
How It Works
DrawVideo decomposes long videos into shots, each defined by a sketch, appearance prompt, and motion prompt. The framework first generates a reference keyframe aligned with the desired structure, then expands this into derivative keyframes that represent various action states.
Finally, it synthesizes clips between adjacent keyframes to create fluid motion, allowing for detailed control over pose, layout, and dynamics.