EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Aug 7·5 min read
99% of My Website Traffic Is Bots
TL;DR
After a year of battling web scrapers on a 1.5 million-page philanthropy donor database, key strategies were developed to mitigate bot traffic effectively.
✦ Why It Matters
Implement effective firewall rules today to reduce bot traffic on your website.
Key Takeaways
- Analyze server logs instead of relying solely on visitor stats.
- Implement country blocks for regions with high bot traffic.
- Use CAPTCHA challenges to filter out bot requests.
- Monitor bot-to-human traffic ratios to adjust security measures.
How It Works
The author utilized Cloudflare's firewall to create specific rules that block or challenge requests based on user-agent strings and geographic locations. By analyzing traffic patterns, they identified and blocked high-volume bot traffic while allowing legitimate users to access the site with minimal friction.