Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
Data Science
Towards Data Science·21d ago·5 min read
How Much Does It Actually Cost to Run a Local LLM? (Euros per Million Tokens, Measured)
TL;DR
A controlled benchmark was conducted to measure the actual electricity costs of running local language models on an RTX 3090 GPU. Surprisingly, five out of eight models were cheaper than cloud-hosted APIs, defying expectations based on model size.
✦ Why It Matters
Engineers should analyze the cost of running local LLMs on their hardware to optimize budget and performance.
Key Takeaways
- Benchmark local LLMs to assess cost-effectiveness.
- Use real-time power tracking for accurate measurements.
- Compare multiple models to find the best value.
- Evaluate your electricity tariffs when calculating costs.
How It Works
The study utilized a controlled environment where three language models were run on the same hardware with a fixed workload. Power consumption was monitored using a custom dashboard that integrated GPU energy usage into cost calculations, allowing for precise measurement of the energy cost per token generated.