We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hugging Face Blog·Jun 17·5 min read
MolmoMotion: Language-guided 3D motion forecasting
TL;DR
Existing models excel at observing motion but struggle with predicting future movements, which is crucial for applications like robotics. MolmoMotion is a new model that forecasts 3D point trajectories based on video frames and action descriptions.
✦ Why It Matters
Engineers can leverage MolmoMotion for improved motion prediction in robotics and video generation applications.
Key Takeaways
- Explore MolmoMotion for improved 3D motion predictions.
- Utilize the MolmoMotion-1M dataset for training your models.
- Evaluate your forecasting models using PointMotionBench.
- Integrate MolmoMotion into robotics for better manipulation tasks.
How It Works
MolmoMotion connects language instructions to visual objects by using a backbone model called Molmo 2. It processes video frames, action descriptions, and 3D point coordinates to predict future trajectories.
The model has two variants: the autoregressive variant predicts coordinates step-by-step, while the flow-matching variant captures uncertainty in motion. This allows for accurate forecasting of object movements in 3D space.