Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
Data Science
Simon Willison's Blog·Sep 7·5 min read
Creepy crawlies
TL;DR
Crawlers are consuming more CPU resources on git.kernel.org than legitimate user access, with 14 CPU cores dedicated to rendering commits as HTML. This highlights a growing issue of abusive web scraping.
✦ Why It Matters
Evaluate your web scraping policies to mitigate resource drain from abusive crawlers today.
Key Takeaways
- Monitor server resource usage to identify crawler impact.
- Implement rate limiting for web scraping activities.
- Consider using CAPTCHAs to deter abusive crawlers.
- Educate users about responsible web scraping practices.
Full Summary
Creepy crawlies, as discussed by Konstantin Ryabitsev, reveal a troubling trend at git.kernel.org, the official Git repository for the Linux kernel. The site is experiencing significant resource drain from abusive web crawlers, which are rendering commits into HTML format.
In fact, the CPU cycles spent on this task exceed those used for all legitimate access, including git clones. At any given moment, 14 CPU cores across five geographically distributed nodes are idle, solely processing these requests.
This situation raises concerns for other platforms, such as Datasette, which also serve numerous crawlable web pages. The implications of this trend suggest a need for better management of web scraping activities to protect server resources.