Cloudflare Started Blocking Undeclared AI Crawlers Today. Half of Their Traffic Was Re-Fetching Pages That Had Not Changed.

Cloudflare announced the deadline on July 1, 2026 and gave the AI industry 76 days. That window closed this morning. As of today, September 15, 2026, Cloudflare's default settings block mixed-use crawlers from any page that hosts ads.

Mixed-use means a crawler that blends traditional search indexing with AI training and agent retrieval, and will not tell you which of those it is doing on a given request. If a bot cannot or will not declare its purpose per fetch, it gets blocked on ad-supported pages by default, and the site owner has to go change a setting to let it back in.

Most of the coverage this week framed this as Cloudflare blocking AI. That is the wrong read and it matters, because the policy is indifferent to whether you are an AI company. It is not indifferent to whether you say so.

What Actually Changed at Midnight

The scope is narrower than the headline suggests and worth stating precisely, because the gap between what changed and what people think changed is where the bad takes come from.

Read the last two rows against the first three. The large publishers with negotiated AI licensing deals, dedicated bot management configs and enterprise contracts are almost entirely unaffected, because they already made a decision and the default does not override a decision. The population that moved today is the free tier and the long tail: personal sites, small blogs, forums, regional news, documentation hosted on a hobby plan, every ad-supported page whose owner has never opened the bot management panel.

So the blast radius of this change is not the New York Times. It is the part of the web that nobody negotiates with, which is also the part of the web that supplies most of the diversity in a training corpus and most of the long-tail answers an agent needs at retrieval time. The web's head was already gated. Today the tail got a default.

The Policy Blocks Ambiguity, Not AI

Here is the part I think is genuinely new. Every prior control on this surface was a control on identity. Robots.txt asks who you are. IP allowlists ask who you are. User agent strings, verified bot programs and signed agent tokens all ask who you are.

This asks what you are doing with this specific fetch, and treats refusal to answer as grounds for denial. That is a different primitive. Purpose has never been a field the web required, and the reason it was never required is that for thirty years a fetch had one purpose: a human was going to look at the page. Bots passed human traffic on the internet for the first time this year, ahead of forecasts that had put the crossover in 2027, and once most requests are machine requests the question of what the machine intends stops being rhetorical and starts being the only useful thing to ask.

The move is elegant because it does not require Cloudflare to have an opinion about AI. It requires the crawler operator to have one, in public, per request, in a form the edge can route on. A crawler that declares itself as search gets search treatment. One that declares itself as training gets whatever the publisher decided training is worth. One that declares itself as agent retrieval on behalf of a specific user gets a third answer. The declaration becomes the unit of negotiation, which means it also becomes the unit of pricing, and the thing that can be priced is the thing that eventually gets a market.

Which Brings Us to the Crawler That Cannot Declare

Cloudflare did not name Google in the July announcement. It described the world's largest search engine as having access to roughly 2x more information than other AI companies, on the grounds that it is difficult to stay discoverable in search without also being used for AI. Nobody had to guess who that was about.

Google's answer has been consistent and is not unreasonable: Google Extended exists, it lets a site opt out of training and out of Gemini Apps and Vertex, and using it does not affect inclusion in Search. That is a real control and more than most providers offer.

The problem is that Googlebot, the flagship crawler, fetches for Search, and Search now includes AI Overviews and AI Mode. One fetch, several downstream consumers, at least one of which is generative. Under Cloudflare's taxonomy that is the textbook definition of mixed use. Google cannot split that crawler without splitting the product, and it cannot declare a single purpose per request without conceding that the purposes are separable, which is the claim its position depends on denying.

I do not expect this to be enforced against Googlebot in any visible way, and I would be surprised if many site owners on the free tier discover they have blocked the crawler that sends them traffic. But the structural point stands independent of enforcement: the policy defines a category, the largest participant in that category is architecturally unable to exit it, and the practical burden lands on the AI companies whose crawlers were already separable and already labeled. Rules that only bind the compliant are a familiar shape in this industry. We wrote the same sentence about distillation advisories six days ago.

The Number Nobody Quoted: 50 Percent

Buried in Cloudflare's own data is the statistic I keep coming back to. More than 50 percent of crawl traffic from AI crawlers is spent re-fetching pages that have not changed.

That is not a policy finding. That is an engineering finding, and it is damning in a specific way. Conditional requests have been in HTTP since 1997. A crawler that stores an ETag and sends If-None-Match gets a 304 and transfers nothing. A crawler that tracks Last-Modified and sends If-Modified-Since gets the same. These are not exotic optimizations, they are the default behavior of every serious crawler built in the last twenty-five years, and roughly half the AI crawl volume on one of the largest networks on earth is apparently not doing it.

Which reframes the entire publisher grievance. A large share of the load publishers have been absorbing and describing as extraction is not extraction at all. It is waste. Nobody got a single additional token of training data out of it. The bandwidth was burned, the origin was hit, the carbon was spent, and the crawler re-read a page it already had. If the AI companies fixed conditional requests tomorrow, the volume complaint would shrink by half and the actual dispute, which is about compensation for content that does get used, would be easier to see.

It also tells you something about how these fleets got built. A crawler that re-fetches unchanged pages at that rate was written by someone optimizing for corpus freshness with bandwidth treated as free, which it is, to them. Cloudflare has now made it not free. That is probably the fastest-acting part of this policy and nobody is covering it.

Pay Per Crawl Became Pay Per Use, and the Meter Moved

The second half of the July announcement got less attention than the block. Pay Per Crawl, the marketplace Cloudflare launched in 2025 that let sites charge bots for scraping, is evolving into Pay Per Use. Publishers get paid when their content creates value rather than when it is fetched. The launch partners are Ceramic.ai and You.com: a publisher who opts in is paid when their content appears in Ceramic's AI search results, or when You.com accesses a piece of their premium content.

Economically this is the right model. Paying per fetch prices the wrong event, rewards volume, and as the 50 percent number shows, would have had publishers billing for redundant re-reads. Paying when content is actually used in an answer is closer to what everyone claims to want.

But look at where the meter physically sits.

A crawl is a packet. Cloudflare sees every one of them, counts them, and can bill on a number the publisher can audit against their own logs. A use is a decision made inside a retrieval stack the publisher will never see, by a ranker whose scoring is proprietary, at a moment no third party observes. Cloudflare can tell you a bot fetched your article. It cannot tell you the model read it, weighted it, cited it or discarded it.

So Pay Per Use is a better economic model that runs on a self-reported number. That is the same structure as every other governance mechanism that showed up this month. Embedded evaluators verify a lab's safety practices with access granted by the lab. Threat intelligence reports document misuse using telemetry only the accused platform holds. Capability threshold claims rest on evaluations the model owner runs and scores. Now content compensation rests on usage counts the payer produces.

None of that is evidence of bad faith and I am not alleging any. It is a structural observation about a year in which the AI industry generated an impressive number of accountability mechanisms, almost all of which terminate in the accountable party's own instrumentation.

Our Take

The number that matters is 76: the days between announcement and enforcement. That is an unusually generous runway for a default change, and it was generous on purpose, because Cloudflare was not trying to break anyone. It was trying to get crawler purpose declared. Matthew Prince said as much, that the hope was to encourage mixed-use crawlers to separate search from agent use and training. The block is the enforcement mechanism for a schema change.

And the schema change is the real story. A private company that sits in front of a large fraction of the web has unilaterally decided that machine requests must carry a purpose field, and has attached a penalty to omitting it. No standards body voted. No regulator ruled. There was no RFC. One vendor changed a default, and starting this morning a meaningful share of the web requires AI crawlers to state intent as a condition of entry.

That is the most consequential governance action of the past week, and it was taken by an infrastructure company rather than by any of the parties who spent the weekend arguing about pacing the frontier. The pacing debate produced three proposals, one of which binds one company unilaterally and two of which require parties who have not agreed. This produced a default that flipped on schedule. I know which one changed more behavior today.

The caveat I would attach: a purpose declaration is an assertion, not a proof. A crawler that says it is doing search retrieval and is in fact building a training corpus is indistinguishable at the edge from one telling the truth. The declaration layer works on reputation and contract, not on verification, which puts it in the same category as everything else described above. It is still a large improvement over a world with no field at all, because you cannot enforce against a claim nobody is required to make.

Practical Read for Builders

If you operate a crawler or an agent that fetches live web content, this is the week to audit two things. First, whether your fetches declare purpose in a form Cloudflare's edge recognizes, because an undeclared agent retrieval on behalf of a real user is now blocked by default on a class of pages it previously reached. Second, whether you send conditional requests. If your fleet is anywhere near the 50 percent re-fetch figure, that is free bandwidth savings and free goodwill, and it is a half-day of work. Our crawler access tracker covers which providers publish separable bots and what they declare.

If you run a site, check whether you are in the population that moved. Free tier, or a zone created recently, plus ads on the page, means your default changed this morning without you doing anything. That may be exactly what you wanted. It may also mean an agent a reader asked to summarize your article is now getting a 403, which is a different outcome from blocking a training scrape and worth deciding on deliberately rather than inheriting. The agent-ready checklist walks the settings.

If you publish and are evaluating Pay Per Use, ask the question the announcement does not answer: what does the usage report look like, how often does it arrive, and what can you reconcile it against. A revenue share on a number you cannot audit is a relationship, not a contract. Ceramic.ai and You.com are small enough to be motivated to get this right, which is why they are the launch partners; the terms that matter are the ones a large provider offers on the same rails next year. We track the licensing side on data licensing.

Three Signposts

Whether any major AI provider ships a genuinely split crawler fleet with separate published identifiers for search, training and agent retrieval, and declares purpose per request rather than per bot. That is the outcome the policy was designed to produce. A provider renaming an existing bot and declaring the same purpose on every fetch satisfies the letter and does nothing, and the difference will be visible in the logs within a month.

Whether Googlebot ends up inside or outside the mixed-use category in practice. If Cloudflare carves out the crawler it spent a paragraph describing as having 2x everyone else's access, the policy becomes a tax on Google's smaller competitors, which is the opposite of its stated intent. If it does not carve it out, we find out how many free tier site owners will accept a search traffic risk to enforce an AI content position.

Whether Pay Per Use publishes an audit mechanism. Not a dashboard, an audit mechanism: something a publisher or a third party can use to check a usage count against an independent signal. If one appears, this becomes the first content compensation scheme in the AI era with a verifiable denominator. If none appears within two quarters, Pay Per Use is a good idea running on trust, and we will be writing the same paragraph about it that we have written about every safety self-report this month.