We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hacker News·25d ago·5 min read
Show HN: We beat Cloudflare's bot detection (open-source stealth browser)
TL;DR
Cloudflare's default blocking of AI crawlers led to agents misreporting data by reading error pages instead of actual content. A new open-source stealth browser was developed to address this issue by accurately detecting and bypassing these blocks.
✦ Why It Matters
Engineers can implement the open-source stealth browser to improve data accuracy when scraping websites protected by Cloudflare.
Key Takeaways
- Measure how often your agents misread block pages.
- Implement the open-source stealth browser for accurate data fetching.
- Monitor the performance of your AI agents post-implementation.
- Stay updated on changes in web security measures affecting data scraping.
Full Summary
As of July 1, 2025, Cloudflare began blocking AI crawlers by default for new domains, resulting in the rejection of hundreds of billions of bot requests. This change caused issues for AI agents that would confidently report incorrect information, often reading Cloudflare's 'Just a moment' block page instead of the intended content.
To tackle this, a new open-source stealth browser was created, designed to detect when a fetch fails due to such blocks. The methodology involved measuring the frequency of these misreads and implementing a solution that accurately identifies and bypasses the block pages.
The findings showed a significant reduction in incorrect data reporting, enhancing the reliability of AI agents. This development has implications for data scraping and AI training, as it allows for more accurate data collection from websites protected by Cloudflare.