EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Aug 17·5 min read
Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP
TL;DR
A dense 27B model was optimized for local inference on an NVIDIA RTX PRO 4000, achieving a throughput of 50.44 tokens per second. The experiment highlighted that the best performance often comes from careful integration rather than just selecting the highest-performing components.
✦ Why It Matters
Consider optimizing your model integration rather than just upgrading hardware for better performance.
Key Takeaways
- Focus on system integration for optimal performance.
- Experiment with different quantization methods for better results.
- Monitor GPU memory usage to avoid out-of-memory errors.
- Use custom builds of llama.cpp to enhance throughput.
How It Works
The Qwen3.8-27B model utilizes a combination of Gated DeltaNet layers and full-attention layers, optimizing memory layout and workload distribution across two GPUs. The embedded MTP allows for efficient token prediction, significantly increasing throughput by leveraging the GPU's peak memory bandwidth.