Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 27·6 min read
Towards Feedback-to-Plan Decisions for Self-Evolving LLM Agents in CUDA Kernel Generation
TL;DR
LLMs generate CUDA kernels (GPU code) through iterative feedback loops, but it's unclear how different feedback types influence planning decisions. CUDAnalyst was built to isolate and measure individual feedback contributions without confounding effects from trajectory drift.
✦ Why It Matters
Engineers can now identify which feedback signals most improve CUDA kernel generation, enabling targeted optimization of LLM agent training and deployment.
Key Takeaways
- Use CUDAnalyst to isolate individual feedback contributions in iterative code generation without confounding effects.
- Design feedback loops prioritizing high-impact signals identified through causal attribution analysis.
- Avoid standard ablations for iterative systems; use trajectory-aware decomposition methods instead.
- Apply feedback attribution insights to improve LLM agent planning for GPU code optimization.
Full Summary
CUDA kernel generation—writing optimized GPU code—benefits from LLM agents that iteratively refine solutions based on feedback. However, standard evaluation methods (ablations) fail to isolate which feedback signals matter because early changes compound across iterations, mixing true feedback effects with random trajectory drift.
CUDAnalyst provides a unified analysis framework to decompose heterogeneous feedback signals (compilation errors, performance metrics, correctness checks) and measure their individual causal impact on planning decisions. The tool separates signal attribution from trajectory-dependent noise by controlling for iterative amplification.
Results clarify which feedback types drive kernel improvements and how planning decisions combine multiple signals. This enables engineers to design more efficient feedback loops for code generation agents.