Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 11·5 min read
Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality
TL;DR
Existing methods for selecting pretraining data for language models often require additional classifiers, which can be computationally expensive and reliant on labeled data. WebGraphMix is a new framework that calculates structural centrality scores from the Common Crawl web graph to adjust the mix of central and peripheral documents.
✦ Why It Matters
Engineers can implement WebGraphMix to optimize pretraining data selection, improving model performance while reducing computational costs.
Key Takeaways
- Adopt WebGraphMix for efficient pretraining data selection.
- Utilize structural centrality scores to assess document importance.
- Reduce reliance on labeled data in data selection processes.
- Enhance model performance with a balanced document mix.
Full Summary
Pretraining data composition is crucial for the effectiveness of language models, yet many current selection methods depend on auxiliary classifiers that introduce complexity and require labeled data. WebGraphMix was developed as a lightweight framework that leverages structural centrality scores derived from the Common Crawl host-level web graph.
By varying the ratio of central (more relevant) to peripheral (less relevant) documents, WebGraphMix optimizes the data selection process. The methodology involves analyzing the web graph to identify document importance based on their connectivity.
Results indicate that using WebGraphMix can lead to improved model performance while reducing computational costs. This approach not only streamlines the data selection process but also opens avenues for more efficient training of language models.
Engineers and researchers can benefit from adopting this framework to enhance their model training efficiency.