Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 25·8 min read
Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most
TL;DR
Larger language models fail at forecasting tasks involving rapid growth and tail risk (extreme outlier events), a pattern common in finance and epidemiology. Researchers created ForecastBench-Sim, a contamination-free benchmark, and tested models on COVID-19, measles, and housing data, finding capable models shift upper-tail predictions upward while ignoring downside risk.
✦ Why It Matters
Engineers deploying LLMs for financial or epidemiological forecasting must use tail-inclusive evaluation metrics and recognize that larger models may produce dangerously miscalibrated extreme-event predictions.
Key Takeaways
- Evaluate LLM forecasts using continuous unbounded metrics, not just binary thresholds, to catch tail miscalibration.
- Test forecasting models on superlinear growth and regime-change scenarios before deployment in finance or epidemiology.
- Larger model capability does not guarantee better distributional forecasts; validate on domain-specific tail risks.
- Decompose forecast errors by quantile to detect upper-tail failures that aggregate metrics may hide.
Full Summary
Language models show inverse scaling—counterintuitively performing worse as they become more capable—on forecasting problems with superlinear growth and tail risk (the possibility of sudden regime changes). Researchers documented this phenomenon across synthetic and real-world datasets including COVID-19, measles outbreaks, housing markets, and hyperinflation.
They introduced ForecastBench-Sim (FBSim), a simulated-world benchmark free from data contamination, and tested Llama-3.1 variants to isolate effects of model scale and post-training. Analysis revealed failures concentrate in upper-tail predictions: larger models aggressively extrapolate growth trends upward while leaving lower-tail estimates unchanged, producing poorly calibrated probability distributions.
Critically, single-threshold metrics (binary pass/fail cutoffs) commonly used in LLM benchmarks show opposite results, masking the real cost. Domain knowledge did not reliably improve calibration.
The authors recommend continuous, unbounded accuracy measures alongside threshold-based metrics for forecasting evaluation.