NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·1h ago
AI Research
arXiv CS.AI·May 25·4 min read
ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload
TL;DR
Researchers identified a gap in LLM evaluation benchmarks. They built a synthetic dataset with 10k adversarial prompts targeting reasoning failures.
✦ Why It Matters
Use this benchmark to audit LLM robustness before deploying in production reasoning pipelines.
Key Takeaways
- Implement ReCoVer for resilient LLM training in GPU clusters.
- Utilize fault-tolerant collectives to isolate hardware faults.
- Adopt in-step recovery to maintain training progress.
- Redistribute workloads dynamically to optimize resource usage.
How It Works
ReCoVer maintains a constant number of microbatches during each training iteration, which ensures that the gradients computed are stochastically equivalent to those from a failure-free run. This is achieved through three layers: fault-tolerant collectives that prevent faults from affecting other replicas, fine-grained recovery mechanisms that preserve progress within iterations, and a dynamic workload policy that reallocates tasks among available GPUs.