Google is a leader in The Forrester Wave™: Public Cloud Platforms, Q3 2026
cloud.google.com·6d ago
Cloud & Infrastructure
The New Stack·6d ago·5 min read
Why an old caching trick is your secret to lower LLM costs
TL;DR
Implementing a caching strategy can significantly reduce costs when using large language models (LLMs) by avoiding repeated queries for the same information. This approach leverages existing data to minimize expenses associated with LLM usage.
✦ Why It Matters
Start implementing caching for repeated LLM queries to cut costs immediately.
Key Takeaways
- Implement exact-match caching using SHA-256 hashes.
- Use semantic matching for similar but non-identical queries.
- Tune cache freshness based on data type and use case.
- Monitor hit rates to validate caching effectiveness.
How It Works
The caching mechanism involves creating a unique fingerprint for each query using a cryptographic hash. If a query matches an existing cache entry, the stored response is returned immediately, bypassing the model.
For queries that do not match, a semantic search identifies similar queries using vector embeddings, allowing for the reuse of relevant responses. This hybrid approach optimizes both exact and approximate matches, significantly reducing the number of model calls.