EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Aug 21·5 min read
What happens when a GPU reads memory
TL;DR
A detailed exploration of how a GPU, specifically the RTX 4090, processes a global memory read during a vector addition operation reveals the complex journey of data through various memory hierarchies. Key findings include the latency costs associated with each stage, totaling approximately 255 nanoseconds for a complete memory access cycle.
✦ Why It Matters
Optimize memory access patterns in your GPU applications to reduce latency and improve performance.
Key Takeaways
- Understand the latency costs of L1 and L2 cache misses.
- Optimize memory access patterns to minimize global load delays.
- Utilize timing experiments to identify performance bottlenecks.
- Leverage insights from GPU memory hierarchies for better application design.
How It Works
The GPU processes a global load instruction by first resolving addresses from registers, then issuing requests to the L1 cache. If a cache miss occurs, the request is translated and sent to the L2 cache, and if that also misses, it proceeds to DRAM.
Each stage has specific latency costs, with the entire process taking approximately 255 nanoseconds.