We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Reddit r/LocalLLaMA·May 20·5 min read
Qwen 3.6 35B GGUF: NTP vs MTP quantization results across GPUs and CPUs
TL;DR
Running large language models locally requires choosing quantization methods (techniques that reduce model precision to fit on consumer hardware), but performance trade-offs between NTP (Neural Tensor Precision) and MTP (Mixed Tensor Precision) across different hardware weren't well documented. A benchmark tested Qwen 3.6 35B—a 35-billion-parameter open model—under both quantization approaches on GPUs and CPUs, measuring inference speed and memory usage.
✦ Why It Matters
Engineers deploying Qwen locally can now choose quantization methods based on their actual hardware rather than guessing.
Key Takeaways
- Benchmark NTP and MTP quantization on your target hardware before production deployment.
- GPU execution may favor MTP quantization; CPU execution may favor NTP for memory efficiency.
- Quantization method choice significantly impacts inference speed and memory consumption.
- Validate quantization performance across both GPU and CPU if supporting multiple deployment targets.
Full Summary
Deploying large language models locally demands balancing model quality against computational constraints. Quantization—reducing numerical precision of model weights from 32-bit floats to lower bit-widths—enables running models on consumer hardware.
This benchmark compared two quantization strategies on Qwen 3.6 35B, a 35-billion-parameter model: NTP (Neural Tensor Precision), which applies uniform precision reduction, and MTP (Mixed Tensor Precision), which selectively applies different precisions to different layers. Testing across GPU and CPU hardware revealed performance trade-offs: GPU execution showed faster throughput with MTP in some scenarios, while CPU inference favored NTP for memory efficiency.
Results indicate quantization method choice significantly impacts real-world deployment performance and should be validated against target hardware.