Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
Data Science
Towards Data Science·Jul 28·5 min read
How Much Does a Local LLM Actually Cost to Run? I Measured Every Watt on Apple Silicon
TL;DR
A detailed analysis of the energy costs associated with running local large language models (LLMs) on Apple Silicon reveals significant power consumption, with costs measured at $0.31 per kWh. The findings indicate that energy usage is even higher than predictions based on RTX-3090 benchmarks.
✦ Why It Matters
Evaluate your local LLM deployment strategy based on energy costs to optimize expenses.
Key Takeaways
- Use TokenWatt to measure energy costs of your models.
- Select larger mixture-of-experts models for better efficiency.
- Avoid choosing models based solely on parameter count.
- Consider your electricity rate when calculating operational costs.
How It Works
TokenWatt measures energy consumption by forwarding requests to an inference server while capturing energy usage data from the Apple Silicon chip. It calibrates its readings against a smart plug to ensure accuracy, providing reliable cost estimates for running various LLMs.