We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Reddit r/LocalLLaMA·May 19·5 min read
48GB VRAM users, what are your daily drivers? Do you wish you had more VRAM? What would you run if you did?
TL;DR
Users with 48GB of VRAM (graphics memory for running AI models locally) discussed whether their hardware meets their needs for running large language models (LLMs—AI systems that generate text). The conversation revealed practical constraints: most users run models between 13B and 70B parameters (model size measured in billions of weights), with 48GB sufficient for quantized versions (compressed models using lower-precision numbers) but limiting for full-precision inference.
✦ Why It Matters
Engineers deploying local LLMs can benchmark realistic VRAM requirements and quantization trade-offs against their hardware constraints.
Key Takeaways
- 48GB GPUs support 30B–70B parameter models with 4-bit or 8-bit quantization.
- Quantization reduces model precision to fit memory but trades accuracy for speed.
- 80GB+ VRAM enables unquantized inference and multi-model deployment.
- Benchmark your workload: measure actual memory usage before purchasing hardware.
Full Summary
A Reddit discussion in r/LocalLLaMA gathered practitioners using 48GB graphics processing units (GPUs) to share their workflows and hardware constraints. VRAM (video random-access memory) is the dedicated memory on GPUs that stores model weights and intermediate computations; insufficient VRAM forces users to quantize models (reduce numerical precision) or use slower disk-based techniques.
Participants reported running models like Llama, Mistral, and other open-source LLMs at various quantization levels (4-bit, 8-bit precision). The consensus indicated 48GB supports 30B–70B parameter models with quantization but prevents running larger 100B+ parameter models or multiple models concurrently.
Users expressed interest in 80GB or 96GB systems to eliminate quantization trade-offs and enable ensemble or multi-model inference workflows.