Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Sep 4·5 min read
Can LLMs Extract Architectural Design Decisions from Source Code Commits? - A Preliminary Exploratory Study
TL;DR
A preliminary study explores the ability of large language models (LLMs) to extract Architectural Design Decisions (ADDs) from source code commits. While models like Gemini 3 Pro and others show promise, they often produce lengthy outputs that lack the necessary rationale behind decisions.
✦ Why It Matters
Consider integrating LLMs into your code review process to better capture architectural decisions.
Key Takeaways
- Explore LLMs for extracting architectural decisions from code commits.
- Use few-shot prompting to improve model output quality.
- Focus on capturing the rationale behind architectural decisions.
- Consider developing architecture-aware LLM systems for better AKM.
Full Summary
Architectural Design Decisions (ADDs) are crucial for understanding the structure and evolution of software systems, yet they are often undocumented and embedded within source code commits. This study investigates whether large language models (LLMs) can effectively extract these decisions.
Four LLMs—Gemini 3 Pro, DeepSeek R1, Kimi K2, and Qwen3—were tested using zero-shot and few-shot prompting on 30 ADDs from open-source projects. The outputs were evaluated using metrics like ROUGE-L and BERT-F1, with results showing all models achieving a BERT-F1 score above 0.81.
Notably, few-shot prompting improved Gemini's BERT-F1 from 0.828 to 0.847. However, the generated ADDs tended to be overly detailed and implementation-focused, often missing the underlying rationale.
This indicates a need for more architecture-aware LLM systems to enhance automated Architectural Knowledge Management (AKM).