Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·21h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
Unifying Data, Memory, and Compute Efficiency in LLM training: A Survey
TL;DR
Large Language Models (LLMs) face challenges in data, memory, and compute efficiency during training. This survey presents a unified framework that integrates techniques for optimizing these aspects, including data pruning and memory-efficient architectures.
✦ Why It Matters
Engineers can apply these unified techniques to optimize LLM training, reducing costs and resource usage.
Key Takeaways
- Adopt a unified approach to optimize LLM training resources.
- Utilize data selection methods tailored to specific task objectives.
- Address GPU memory limitations by optimizing multiple components together.
- Implement compute-aware strategies to allocate resources effectively.
How It Works
The survey introduces a constraint-centric perspective, emphasizing that data selection, memory management, and compute allocation must be considered together. Techniques such as gradient-based scoring and curriculum learning are discussed for improving data efficiency.
The authors argue that effective scaling requires simultaneous optimization of weight storage and memory usage, rather than focusing on individual components.