Cloudflare and GoDaddy announced a strategic partnership this Tuesday aimed at giving website owners greater control over how AI agents scrape their data. The core of the collaboration involves integrating Cloudflare’s “AI Crawl Control” tool directly into the GoDaddy platform.
This tool grants site administrators the autonomy to allow or block specific AI bots, and even set up paywalls for data scraping. As AI-powered search tools increasingly siphon off internet traffic, this initiative is designed to protect the economic interests of site operators, preventing the loss of clicks caused by content being scraped without compensation.
“By providing website owners with these tools and open standards, we are building the foundation for a new internet business model,” said Stephanie Cohen, Chief Strategy Officer at Cloudflare. Cohen emphasized that the company is committed to ensuring every creator can verify the identity of visitors while providing a secure, transparent path for legitimate AI agents to engage.
Cloudflare has previously taken aggressive steps to limit unauthorized scrapers. In 2025, the company introduced a technology that uses “junk data” to trap and exhaust the computing power of bots. Cloudflare noted that many content publishers have suffered significant revenue losses as tools like Google AI Overviews provide answers directly to users, discouraging them from clicking through to original links.
Establishing an AI Agent Identity System
Beyond blocking unwanted traffic, the two companies are jointly supporting an industry standard designed to distinguish between “good” and “bad” bots. To this end, they have expressed clear support for the “Agent Name Service” (ANS).
Developed by GoDaddy and open-sourced on GitHub, the system functions similarly to the internet’s DNS. By establishing a protocol-agnostic registry, ANS links AI agents to the entities behind them, thereby increasing trust in the online ecosystem.
Furthermore, both companies are promoting Cloudflare’s “Web Bot Auth” verification method. This multi-pronged approach is not just about building a defensive wall; it is an attempt to establish a viable, compliant framework for transactions between content producers and AI developers in the age of artificial intelligence.
