NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·1h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Minibatch Selection via Partition Matroid Constrained Gradient Matching
TL;DR
Training large language models on diverse data faces challenges in selecting minibatches that ensure both fast convergence and domain coverage. PartitionSel is a new approach that optimizes minibatch selection by maximizing a validation-guided gradient-matching utility while adhering to per-domain constraints using a partition matroid.
✦ Why It Matters
Engineers can leverage PartitionSel to enhance minibatch selection, improving training efficiency for large language models on diverse datasets.
Key Takeaways
- Use PartitionSel for efficient minibatch selection in LLM training.
- Leverage partition-matroid constraints to manage domain budgets.
- Reduce redundancy in sample selection across diverse domains.
- Achieve faster convergence with improved compatibility in training updates.
How It Works
PartitionSel optimizes minibatch selection by maximizing a utility function that matches gradients across domains while respecting predefined budgets. This is achieved through a partition-matroid constraint, which ensures that selections from different domains do not overlap excessively, thus enhancing the diversity and effectiveness of the training data.