Starting September 15, 2026, Cloudflare blocks Training and Agent AI crawlers by default on ad-monetized pages. Here's what changed, what it means if you do nothing, and how to decide your settings.
If your website runs behind Cloudflare and shows ads, the rules for which AI bots can access your content may have already changed automatically as of September 15, 2026 — without you touching a single setting.
Starting that day, Cloudflare put new default rules in place for how AI crawlers are treated on any page that shows ads. The new defaults sort bots into three categories — Search, Agent, and Training — and Training and Agent crawlers are now blocked by default on ad-monetized pages, while Search crawlers remain allowed.
What Changed on September 15
Crucially, these new defaults only apply automatically to:
- Brand-new domains onboarded to Cloudflare on or after September 15, 2026
- New sites added by existing Cloudflare customers
- All existing free-plan Cloudflare customers
If you already had settings configured before that date as a paying customer, Cloudflare says nothing changes automatically — your existing choices remain in place. But it’s still worth checking, for reasons we’ll get into below.
Why Cloudflare Did This
Cloudflare CEO Matthew Prince framed the move bluntly in the company’s announcement: “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.” The company is calling the underlying policy shift “Content Independence Day” — the idea that an ad on a page signals the site owner meant a human visitor to land there, so scraping that same page to train a model or power an AI agent is a fundamentally different transaction than a search engine indexing it for referral traffic.
Cloudflare specifically called out what it describes as a “Google asymmetry” problem: Googlebot bundles search indexing together with AI training data collection, meaning publishers who want to keep their content out of model training risk losing search visibility too. Google has pushed back on this framing, noting it offers a separate “Google-Extended” bot that lets site owners opt out of AI training use without affecting Search inclusion — though its main Googlebot crawler still powers AI Overviews and AI Mode as part of standard Search.
The Three Crawler Categories, Explained
| Category | What It Does | Default After Sept 15 (ad pages) | Example |
|---|---|---|---|
| Search | Indexes content to answer questions later; typically sends referral traffic back | Allowed | Classic search-engine crawling |
| Agent | Acts in real time on a specific person’s behalf, right now | Blocked | An AI assistant fetching a page a user just asked about |
| Training | Scrapes content to train a model, with no direct return for that use | Blocked | Bots feeding large language model training runs |
Crawlers that mix these purposes together — “mixed-use” crawlers — are blocked by any setting that blocks Training, including the older, simpler “Block AI Bots” toggle many site owners already had switched on.
What This Means If You Do Nothing
For many existing small business sites already on Cloudflare with settings configured, nothing changes today. But it’s worth actually checking, because:
- If you’re on a free Cloudflare plan, the new defaults do apply to you, not just to brand-new signups
- If you spin up a new site or subdomain on Cloudflare going forward, it will start with Training and Agent blocked by default on any page carrying ads
- If you run any kind of AI chatbot, browsing agent, or MCP server that needs to fetch your own pages on a visitor’s behalf, you may need to explicitly allow the Agent category, or it could get blocked by default
Should You Block, Allow, or Charge?
- Block Training and Agent if: you don’t want your content used to train AI models or accessed by browsing agents at all, and you’re not concerned about losing potential AI-referral traffic.
- Allow Training if: you want your brand and content to show up in AI-generated answers (ChatGPT, Gemini, Perplexity) over time, and you’re comfortable trading some control for visibility.
- Allow Agent if: you run tools, checkout flows, or content that AI assistants need to access in real time on a user’s behalf — for example, an MCP server or an AI-friendly help center.
- Explore Pay Per Crawl / Pay Per Use if: you want to keep Training access open but be compensated when your content is actually used. Cloudflare’s evolving program, currently piloting with partners like Ceramic.ai and You.com, pays publishers when their content shows up in an AI answer, not just when it’s fetched.
How to Check and Adjust Your Settings
- Log into your Cloudflare dashboard for the site in question.
- Go to Security settings and look for the AI bot or crawler management section.
- Check whether “Block AI Bots” or the newer Search/Agent/Training controls are configured, and what each is currently set to.
- Decide, category by category, whether you want each type allowed, based on the decision framework above.
- If you want AI referral visibility, make sure Training access matches that goal — a blanket “Block AI Bots” setting also blocks Training-labeled traffic.
- Recheck your robots.txt file too. Your Cloudflare dashboard settings and your site’s own robots.txt should tell AI crawlers the same story, not contradict each other.
FAQs
Does this affect my Google Search rankings?
Not directly. Search-category crawlers, including the crawler behind standard Google Search, AI Overviews, and AI Mode, remain allowed under Cloudflare’s new defaults. This change specifically targets Training and Agent crawlers.
Will this automatically change my existing website’s settings?
Not if you’re an existing paid Cloudflare customer with settings already configured. It applies automatically only to new domains, new sites from existing customers, and existing free-plan customers.
What’s the difference between an Agent crawler and a Training crawler?
An Agent crawler acts in real time on a specific person’s behalf right now, like an AI assistant fetching a page a user asked about. A Training crawler scrapes content to train a model over time, with no immediate, individual request behind it.
Should I just block all AI crawlers to be safe?
Not necessarily. Blocking Training and Agent crawlers protects your content from being used to train models or accessed by browsing agents, but it may also reduce your chances of being cited or recommended inside AI-generated answers, which is becoming a real source of discovery.
How do I get paid if I allow AI companies to use my content?
Cloudflare’s “Pay Per Use” program, currently piloting with partners like Ceramic.ai and You.com, is designed to compensate publishers when their content actually appears in an AI result, not simply when a crawler fetches the page. It’s still an early-stage program, so availability may vary by site.
Closing
For most of the web’s history, crawling was a simple trade: a bot indexed your page, and in exchange, you got a visitor. AI crawlers broke that bargain by taking content without sending anything back. Cloudflare’s September 15 change doesn’t resolve that tension, but it does something useful: it forces every site owner to make an explicit, informed choice about AI access instead of leaving it to a default nobody chose.
Written by mootkur9
17 years in digital marketing. Founder of Sadhyam. Writing about AI search, AEO/GEO, and the agentic web. More about me →
