NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 25·8 min read
ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse
TL;DR
Large language model (LLM) serving systems cache key-value pairs to avoid recomputing shared request prefixes, but this cache often exceeds GPU and local memory capacity, forcing expensive remote storage. ObjectCache stores KV cache in S3-compatible object storage while minimizing latency by scheduling data delivery to match GPU consumption order.
✦ Why It Matters
Engineers can reduce LLM serving infrastructure costs by shifting KV cache to object storage without sacrificing latency-critical first-token performance.
Key Takeaways
- Store KV cache in object storage (S3, Ceph, DAOS) instead of remote DRAM to reduce cluster costs.
- Schedule data transfer to match GPU consumption order, overlapping network I/O with compute to hide latency.
- Expect 5.6% TTFT overhead for large contexts (64K tokens) but 56–75 ms overhead for small contexts (4K tokens).
- Use adaptive bandwidth scheduling to achieve 1.2–1.8x better TTFT under shared bandwidth constraints.
Full Summary
LLM inference systems use prefix KV caching—storing precomputed key-value tensors from shared input prefixes (like system prompts)—to reduce time-to-first-token (TTFT, the latency before generating the first output token). However, accumulated cache often exceeds GPU memory and local DRAM capacity, forcing systems to store it in remote DRAM pools, which increases cluster size and operational cost.
ObjectCache proposes storing KV cache in S3-compatible object storage (like Ceph or DAOS), eliminating capacity constraints. The key innovation is co-designing the storage protocol and transfer scheduler so data arrives in the order the GPU consumes it, overlapping network transfer with GPU computation across concurrent requests.
Prototyped on a 100 Gbps RoCE cluster with NIXL inference library, Ceph RGW, and DAOS, ObjectCache achieves 5.6% latency overhead for 64K-token contexts and 1.2–1.8x better TTFT under bandwidth constraints compared to equal sharing.