Cloudflare changed the rules for AI: What it means for your website

Pay-Per-Crawl is Cloudflare’s new licensing programme that allows AI companies to pay for structured access to web content. Rather than scraping content without permission, this model encourages negotiation and compensation.

Cloudflare is also rolling out new tools to allow websites to customise how different types of AI crawlers interact with their sites, including the ability to allow, block, or limit AI bots independently of traditional search engine crawlers like Googlebot or Bingbot.

Historically, webmasters have relied on robots.txt to control bot behaviour. Publishers would politely signal which areas of their site should be excluded from crawling, and most well-behaved bots would comply. But as AI companies race ahead, many aren’t following these directives reliably, and the impact is measurable. 

According to the Cloudflare crawlers study, AI bots crawl vast amounts of content while sending back almost no referral traffic. 

• GPTBot, OpenAI’s crawler, saw requests increase by +305%, now accounting for 7.7% of all crawler activity.
• The ChatGPT-User agent, spiked by +2,825%, reflecting the wider integration of browsing capabilities in AI tools.
• PerplexityBot grew by an astonishing +157,490%, indicative of its emerging presence and ambitions to rival major AI platforms.

Source: Cloudflare Radar

While Googlebot still leads overall (making up 50% of crawl activity), the crawl-to-referral ratio for AI bots is alarmingly one-sided. 

In short: AI bots take far more than they give back.

This unlocks the potential to monetise high-value content being ingested by generative AI platforms that was previously scrapped for free.

Control is being redefined in AI search

With search now embedded in tools like ChatGPT, Perplexity, and Gemini, brands and publishers must rethink how they control visibility and access to their content. This is a technical SEO and strategic decision.

The value of content is rising 

Your content is no longer just about ranking; it’s now an asset in the broader AI ecosystem. Cloudflare’s shift confirms that high-quality content has tangible value and should be protected, licensed, and rewarded accordingly.

AI crawler visibility is the next SEO challenge

Deciding whether and how to allow AI bots is going to become part of modern SEO strategy. We expect AI bot directives to feature in audits, dashboards, and commercial conversations moving forward.

Cloudflare’s move signals that the open web isn’t just a resource to be mined, it’s a marketplace, and creators deserve a say in how their work is used.

It’s time we took that seriously, it might be the beginning of accountability.

If you’re unsure how AI bots are currently accessing your site; or whether you should allow them at all, our team at Varn can help you audit your site and set up the right controls with our AI Visibility Framework.