EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Sep 4·5 min read
Grep beats LSP? Why coding agents ignore your fancier tools
TL;DR
A comparison of grep and LSP-backed semantic navigation revealed that coding agents often prefer grep despite its simplicity. This suggests that a tool's compatibility with language models may be as crucial as its advanced features.
✦ Why It Matters
Consider prioritizing tools that integrate well with language models over those with advanced features.
Key Takeaways
- Prioritize tools that provide immediate context for coding tasks.
- Test new tools within the full agent workflow before adoption.
- Consider the noise level of codebases when choosing retrieval methods.
- Maintain familiarity in tool interfaces to improve agent efficiency.
Full Summary
In a comparative study, grep, a lexical search tool, was pitted against LSP-backed semantic navigation for code retrieval tasks. The research involved three Claude models and various Python and TypeScript repositories.
Surprisingly, agents chose grep for simple code-location tasks only 0% to 6% of the time, while LSP was favored 45% to 57% for reference-completeness tasks. However, LSP's precision did not translate to better overall performance due to higher token usage and the need for additional actions to interpret results.
The study emphasizes that a tool's effectiveness is not solely based on precision but also on how well it integrates into the agent's workflow. The results suggest that interface design and the context provided by tools are critical for optimizing coding agent performance.