Predict before you train: Scaling Laws for particle physics foundation models

TL;DR

A new scaling law for transformer models in particle physics allows for accurate predictions of model performance before training, achieving predictions within one percent. This enables researchers to translate compute budgets into expected physics outcomes effectively.

Key Takeaways

• Apply the new scaling law to estimate model performance early.
• Use pretrained models to save on training costs.
• Leverage lower pretraining loss for better downstream results.
• Access released training recipes and code for implementation.