NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Scaling Neural Network Verification with Tensor Parallelism and Fully Sharded Data Parallelism
TL;DR
Formal verification of neural networks, which ensures safety across all inputs, is limited by GPU memory constraints. To address this, the authors adapted tensor parallelism and fully sharded data parallelism techniques for the auto_LiRPA and α,β-CROWN verification frameworks.
✦ Why It Matters
Engineers can now verify larger neural networks for safety, enhancing reliability in critical applications.
Key Takeaways
- Implement Tensor Parallelism to reduce memory usage by up to 50%.
- Use Fully Sharded Data Parallelism for an 80-90% drop in baseline memory.
- Validate models on benchmarks like MNIST and CIFAR-100 for effectiveness.
- Focus on optimizing per-neuron alpha tensors for future improvements.
How It Works
Tensor Parallelism divides weight and relaxation-coefficient matrices across multiple GPUs, allowing for more extensive model verification without exceeding memory limits. Fully Sharded Data Parallelism focuses on sharding only weight matrices, using an AllGather operation per layer to maintain accuracy while significantly reducing memory requirements.