Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
Security
Cloudflare Blog·Sep 1·5 min read
How we could save petabytes of cache storage with Zstandard and Pingora
TL;DR
Cloudflare developed a Cache Transcoding system that uses Zstandard compression to significantly reduce cache storage needs, achieving an average asset size reduction to one-third. This approach trades a minor CPU increase for substantial savings in storage and bandwidth across data centers.
✦ Why It Matters
Consider implementing Zstandard compression in your caching strategy to optimize storage and bandwidth usage.
Key Takeaways
- Use Zstandard to compress eligible cache assets.
- Target compressible text formats for maximum efficiency.
- Evaluate the trade-off between CPU cost and storage savings.
- Consider implementing Cache Transcoding in your caching strategy.
How It Works
Cache Transcoding encodes eligible responses using Zstandard before writing them to disk. This compressed form is maintained while the asset is cached and transferred between data centers, only being decoded when served to clients.
The system checks for eligibility based on response type and size, ensuring that only beneficial assets are compressed.