NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
Cloud & Infrastructure
The New Stack·Jul 1·5 min read
“You Only Compute Once”: How Clockwork wants to put an end to AI training restarts
TL;DR
In large GPU clusters, failures often require rolling back to the last checkpoint, which is slow and costly. Clockwork has developed TorchPass, a fault tolerance tool that allows training jobs to continue by migrating their in-memory state to a healthy GPU.
✦ Why It Matters
Engineers can leverage TorchPass to minimize downtime during AI training, improving overall productivity.
Key Takeaways
- Implement TorchPass to avoid costly checkpoint rollbacks.
- Utilize the YOCO Guarantee for reliable AI training.
- Set aside idle GPUs for quick job migration.
- Monitor GPU health to preemptively migrate jobs.
How It Works
TorchPass allows for live migration of training jobs by moving the in-memory state of a job to a spare GPU when a failure is detected. This process can occur even before a failure happens, based on warning signs like temperature thresholds.
The tool operates in two modes: model-aware, which requires minimal code changes for faster recovery, and model-transparent, which is easier to implement but takes longer to recover due to moving more data.