Gemini 4 Carbon: Our next frontier model
deepmind.google·5d ago
AI Engineering
Reddit r/LocalLLaMA·May 20·5 min read
RTX 5080 16GB: Qwen3.6 35B MoE at 128k context — 56 tok/s, and why MTP doesn't help
TL;DR
Running large language models locally on consumer GPUs is constrained by memory and speed limits. A benchmark tested Qwen 3.6 35B MoE (a 35-billion-parameter model using mixture-of-experts, where only relevant sub-networks activate per token) on an RTX 5080 16GB GPU at 128k context length (processing 128,000 tokens of input history).
✦ Why It Matters
Engineers optimizing local LLM inference can expect realistic throughput ceilings and know that multi-token prediction doesn't accelerate MoE models on consumer hardware.
Key Takeaways
- MTP does not improve throughput on RTX 5080; memory bandwidth is the limiting factor, not token generation.
- Qwen 3.6 MoE achieves 56 tokens/second at 128k context on RTX 5080; use this as a baseline for consumer GPU deployments.
- Mixture-of-experts routing reduces computation versus dense models; prioritize MoE architectures for memory-constrained inference.
- At 128k context, memory efficiency matters more than speculative decoding; optimize quantization and context compression instead.
Full Summary
Running large language models locally on consumer hardware requires balancing model size, context window (the amount of prior text the model can reference), and inference speed. Qwen 3.6 35B MoE is a 35-billion-parameter model using mixture-of-experts architecture, where different subnetworks activate selectively based on input, reducing computation versus dense models.
The benchmark measured inference on an RTX 5080 with 16GB VRAM at 128k context length (128,000 tokens of prior text), achieving 56 tokens per second throughput. Multi-token prediction (MTP) is an optimization technique that predicts multiple output tokens in a single forward pass to reduce latency; testing showed MTP provided no measurable benefit in this scenario.
This suggests memory bandwidth or GPU utilization, not token generation count, is the bottleneck. Engineers targeting consumer hardware should prioritize memory-efficient architectures and context optimization over speculative decoding techniques.