Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·15d ago·5 min read
When to Use Which? Benchmarking Optimisers for Configurable Systems under Varying Budgets
TL;DR
Configurable systems often require optimization under budget constraints, presenting a challenge for engineers. This study benchmarks various optimization algorithms to identify their effectiveness across different budget scenarios.
✦ Why It Matters
Engineers can choose the most effective optimizer for their budget constraints, improving project outcomes immediately.
Key Takeaways
- Use model-based optimisers for tight budgets.
- Opt for model-free optimisers when resources are ample.
- Consider FLASH for consistent performance across budgets.
- Evaluate local optima characteristics in your systems.
Full Summary
Software configuration tuning is essential for optimizing system performance, yet the time required for tuning varies significantly across different systems. This study evaluates eight established optimisers across 22 configurable systems, focusing on their performance under varying budget levels.
Results indicate that model-based optimisers, such as SMAC, are effective for tight budgets, while model-free optimisers, like Genetic Algorithms (GAs), perform better with more generous budgets. Interestingly, FLASH, a greedy optimiser, consistently delivers strong results regardless of budget constraints.
The research highlights that many systems have favorable local optima, which allows greedy methods to excel. This work provides valuable insights for software engineers in selecting the right optimiser based on their budget.