Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
Security
Cloudflare Blog·Aug 21·5 min read
Say it once: introducing Bot Preference SynC
TL;DR
Cloudflare has launched Bot Preference Sync, a feature that simplifies managing bot traffic preferences by automatically updating robots.txt based on user configurations. This tool helps website owners better control AI traffic and protect their content.
✦ Why It Matters
Start using Bot Preference Sync today to better manage your website's AI traffic preferences.
Key Takeaways
- Enable Bot Preference Sync to automate robots.txt updates.
- Customize your bot preferences for Search, Agent, and Training traffic.
- Review and confirm your settings during the transition to Bot Preference Sync.
- Utilize the public bots directory for transparency on bot behavior.
How It Works
Bot Preference Sync automatically generates or updates the robots.txt file based on the AI bot preferences set in the Cloudflare dashboard. This ensures that the preferences for allowing or disallowing bot access are consistently enforced, reducing the risk of crawlers ignoring site owners' wishes.