EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Sep 1·5 min read
Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
TL;DR
Qwen3.8-Flash-Next, a 104GB AI model, can be run on a 48GB Mac by streaming from SSD, achieving ~12 tokens per second. This setup allows users to utilize large models without needing extensive RAM.
✦ Why It Matters
Consider using slotstream to run large AI models on limited hardware today.
Key Takeaways
- Ensure at least 110GB of free disk space for model weights.
- Use slotstream to run large models on limited RAM.
- Monitor performance metrics to optimize memory usage.
- Verify downloads to ensure model integrity before use.
How It Works
Slotstream allows the Qwen3.8-Flash-Next model to run on machines with less RAM by streaming model weights from an SSD. It dynamically adjusts memory usage based on available resources, optimizing performance while maintaining responsiveness.