Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 17·5 min read
Building AI-Intensive Software with AI: Early Results and a Cautionary Tale on Measuring Development Cost
TL;DR
A student team developed a conversational onboarding assistant using AI tools, initially reporting a 19.4x cost ratio, later corrected to 9.9x due to measurement errors. This highlights the challenges in accurately assessing development costs in AI-intensive projects.
✦ Why It Matters
Ensure accurate cost assessments in your AI projects by reviewing your measurement methodologies today.
Key Takeaways
- Implement a three-layer cost model for accurate assessments.
- Review pricing structures to avoid common measurement errors.
- Use correct regional labor rates in cost calculations.
- Consider the implications of AI assistance on development costs.
How It Works
The team utilized a three-layer cost model to assess development expenses, which included real AI spending, self-reported human effort, and a counterfactual scenario to estimate what costs would have been without AI assistance.
⚠ The Catch
Errors in calculating per-token costs and using incorrect labor rates can significantly distort cost assessments, leading to inflated ratios that misrepresent the true development expenses.