New Cloudflare AI bot controls: What will the impact be on your SEO?

Websites using Cloudflare are in danger of dropping off the search radar with a significant update in the pipeline.

On 1st July, 2026, Cloudflare marked its second annual Content Independence Day by giving website owners three different levels of control for AI traffic. The straightforward logic is that website owners will be able to block their website from being crawled by AI bots, and therefore prevent their website’s content from being used in LLMs in the future. This sounds like a great change for certain websites, but the way it’s implemented can be extremely damaging to your SEO, if it’s not managed correctly.

Starting from 15 September 2026, Cloudflare’s new default setup will block ‘Training’ and ‘Agent’ bots on pages that contain ads.

However, multi-purpose crawlers like Googlebot, BingBot, and Applebot handle both search indexing and AI training; they don’t do these things with separate robots.

So, if you don’t act, the default setting will kick in and block training bots, you could accidentally block Google and Bing from indexing your site entirely.

For the vast majority of website owners, Cloudflare is a set it and forget it piece of infrastructure. Once activated and set up with basic firewall rules, most people leave Cloudflare alone and don’t need to login to check anything. It would therefore be really easy to simply think this was another update that you could ignore, and assume that everything will continue as normal.

In traditional search, the process is simple: Search engines crawl your site, and in exchange, they send you referral traffic.

In AI search, AI models crawl your site, absorb your content, and answer the user’s question directly on ChatGPT or Google’s AI Overview – potentially without ever sending a click your way.

Cloudflare’s update is a response to this existential threat, giving website owners control over how their content is used, splitting AI behavior into three distinct buckets:

1. Search: Bots that identify content to help answer a user’s query in more detail (and hopefully send referral traffic).
2. Agent: Real-time bots acting directly on behalf of a human user (like a chat assistant retrieving specific data or an agentic shopping bot).
3. Training: Crawlers that are used to absorb your content to train or fine-tune their LLMs.

Depending on your type of business, you might want to block particular types of AI bot, if you don’t want your content seen or referenced in certain places.

The question a lot of clients are asking is, should they use these new features to block AI bots? The decision isn’t black and white, and really depends on your business type and your marketing strategy.

• Guarding your IP: If you create proprietary research or unique journalism, blocking training bots stops tech giants from plagiarising your value proposition to power their own platforms.
• Saving server resources: AI bots are notoriously aggressive and crawl frequently. Blocking them preserves your server bandwidth, lowers infrastructure costs, and keeps your site fast for real human users.
• Pay per crawl opportunity: By blocking free access, you position your brand to participate in emerging ‘Pay-Per-Crawl’ models, meaning AI crawlers will have to pay to access your pages.

• Blocking multi-purpose bots: Blocking ‘Training’ bots will also inadvertently catch multi-purpose bots like Googlebot that do both Training and Search. If Cloudflare blocks Googlebot because it’s a ‘Training’ bot, it also stops Googlebot from indexing your site for traditional search – and your organic traffic would be seriously impacted.
• Missing out on agentic revenue: The web is moving rapidly toward an ’agentic internet’, where AI agents do things for humans, such as product research, price comparisons, and physically making the purchase. If you block bots, you prevent digital assistants from presenting your products or services to motivated buyers.

We’re obsessed by AI search and love all of the new technologies and features that we have available to us. But, some features are dangerous if they’re not implemented correctly.

We have already audited and covered this for our clients. We recognised early on that a bad or default Cloudflare setup with these new AI bot guidelines could be a recipe for an SEO and GEO disaster. You cannot treat AI traffic with a blunt on/off switch – which setup is right for you really depends on your business, your website structure, and your marketing goals.

For our clients, we’ve implemented configurations that allow for maximum visibility in traditional search engines and AI search summaries, depending on the client-specific requirements. Every business is different, and while we typically recommend keeping access open, that’s not the right approach for everyone.

Our clients are ready for this Cloudflare setup change, but it will be an interesting few months while we see what other website owners have decided to do.

Cloudflare’s new default settings for AI crawlers roll out on 15 September 2026. If your site isn’t properly configured, you risk unintentionally blocking your content to AI training or blocking your pages to search engines altogether.

We can help you navigate this setup change, and optimise your digital presence for the future of search, SEO and GEO. Depending on your business needs, we can support your AI Visibility through three core offerings:

• Cloudflare AI crawler setup: Support with the appropriate setup for Cloudflare to ensure you’re only blocking the robots that shouldn’t have access.
• Our AI Visibility Framework: A comprehensive strategy to define exactly which AI bots can access your site, ensuring you get maximum referral traffic without losing your intellectual property.
• AI Presence Reporting: Our reporting solution that shows you exactly how you’re performing in AI search, per topic, per LLM.
• Agentic Shopping Readiness Audit: For e-commerce and B2B brands, we ensure your site architecture allows transactional AI Agents to browse, compare, and purchase from you seamlessly without getting tripped up by security firewalls.
• AI / GEO Training: We provide training sessions, half day or full day workshops, or full AI search training programs, depending on the requirements of your team.

If you’re at all unsure about whether your existing setup is optimal, contact us today to audit your Cloudflare setup and make sure you’re on the right path.