We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hacker News·Jul 3·5 min read
Every AI Visibility Tool Is Lying to You
TL;DR
AI visibility tools claim to measure brand presence in AI responses, but their precision is often misleading. These tools typically scrape data from consumer applications or use APIs, which do not provide a complete picture.
✦ Why It Matters
Engineers should critically evaluate AI visibility tools to ensure they understand the limitations of reported metrics.
Key Takeaways
- Scrutinize the methodology behind AI visibility metrics.
- Understand that scraping provides limited insights into user experience.
- Look for transparency in how visibility data is calculated.
- Use directional signals from tools, but avoid over-reliance on precise metrics.
Full Summary
A gap exists in accurately measuring brand visibility within AI platforms like ChatGPT and Claude, as current tools often present misleading metrics. These AI visibility tools, which include metrics such as mention rate and share of voice, typically rely on scraping the frontend of applications or calling APIs, capturing only a single session's output.
This approach fails to account for the variability and personalization inherent in AI responses, leading to potentially inflated claims about visibility. For instance, a tool might report a brand's visibility as 17% compared to a competitor's 31%, but this figure lacks context regarding how it was derived.
Engineers should be cautious, as these tools often do not disclose their methodologies, making it difficult to assess the reliability of the data. Understanding the limitations of these tools can help engineers and researchers better interpret the results and make informed decisions.