Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·1d ago
AI Research
arXiv CS.AI·Jun 11·5 min read
Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality
TL;DR
Existing methods for selecting pretraining data for language models often require additional classifiers, which can be computationally expensive and reliant on labeled data. WebGraphMix is a new framework that calculates structural centrality scores from the Common Crawl web graph to adjust the mix of central and peripheral documents.
✦ Why It Matters
Engineers can implement WebGraphMix to optimize pretraining data selection, improving model performance while reducing computational costs.
Key Takeaways
- Utilize WebGraphMix for efficient data selection without labeled data.
- Balance central and peripheral documents for improved model performance.
- Explore web graph topology as a new data curation strategy.
- Integrate structural scores with quality classifiers for better outcomes.
How It Works
WebGraphMix computes structural centrality scores from the Common Crawl web graph, identifying central hosts that provide broad knowledge and peripheral hosts that offer niche insights. By varying the ratio of these document types in the training data, the framework enhances the model's ability to generalize and reason across diverse tasks.