We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hacker News·21d ago·5 min read
LeMario: Training a JEPA World Model on Super Mario Bros
TL;DR
A Joint-Embedding Predictive Architecture (JEPA) was built to learn world dynamics in Super Mario Bros. While it successfully predicted short-term outcomes and navigated to nearby goals, it struggled with distant objectives.
✦ Why It Matters
Engineers should consider integrating more complex decision-making strategies alongside predictive models to enhance performance in dynamic environments.
Key Takeaways
- Explore the limitations of predictive models in dynamic environments.
- Integrate decision-making strategies with predictive architectures.
- Test models on varied goal distances to assess robustness.
- Document development challenges to inform future projects.
Full Summary
A Joint-Embedding Predictive Architecture (JEPA) was implemented to explore world dynamics in Super Mario Bros, inspired by its original use in reward-free planning for Push-T. The model was trained from scratch, focusing on predicting future game states based on pixel inputs and actions.
Initial tests showed promising results, with the model generalizing well to unseen episodes and accurately predicting five-step futures. However, when tasked with navigating to more distant goals, the model failed to overcome basic obstacles, indicating it had not truly learned to progress through the game.
This post serves as both a technical walkthrough and a reflection on the challenges faced during development, revealing critical insights about the limitations of predictive learning in complex environments. The findings suggest that while prediction accuracy is important, it does not guarantee effective decision-making in dynamic settings.