AI Crawlers: Which Ones to Block Before September 15, 2026?

You have thirteen days, and you probably do not know which AI bot earns you what. On September 15, 2026, Cloudflare changes its default behavior and starts blocking a subset of AI crawlers on ad-supported pages, without you deciding anything. If your site sits on an existing free account, or if it is a new site under an existing client account, the switch happens on its own. Thousands of sites are about to change their crawl policy blind. The real question is not “should I block AI crawlers.” It is: which ones, and on what data. Here is the method to decide, logs in hand, before the default decides for you.

What changes on September 15, 2026

The facts, without the drama. Cloudflare announced on July 1, 2026 that from September 15, training and agent crawlers would be blocked by default on pages monetized with advertising. The change targets three groups: new domains, new sites on existing accounts, and above all every existing free account that has not touched its settings. Search crawlers stay allowed by default.

Two details matter more than the headline. First, “mixed” bots like Googlebot, which crawl for both search and training, are handled by the most restrictive rule: a setting that is too broad can push you out of Google without you meaning to. Second, Cloudflare is moving its 2025 “Pay Per Crawl” toward a “Pay Per Use” model: you get paid when your content shows up in an AI answer, not just when a bot fetches the page. The principle is better, but it does not excuse you from knowing what each bot does on your site.

The three families of AI crawlers, and why the confusion is expensive

This is mistake number one, and it is everywhere. People talk about “the AIs” as a single block, when there are three jobs behind them, with three opposite effects when you block them. Blocking GPTBot does not remove you from ChatGPT answers. Blocking OAI-SearchBot does. Until that point is clear, any blocking decision is a coin flip.

Remember the logic, not the table by heart. The training bot can be blocked with no pain for your traffic (it is a content ownership call, not a visibility one). The search index bot is the one that puts you in the answers, or not: blocking it means giving up the citation. The live fetch bot is triggered by a real user, so blocking it means shutting the door on someone actively looking for you. Do not confuse it with a human-driven agentic browser, which renders the page in a real browser: that case is covered in AI Browsers in GA4: How to Detect Atlas and Comet Sessions, and it is not handled at the crawl level.

Why GA4 is blind here

Your first instinct will be to open GA4 to see “how much is coming through.” Do not count on it: GA4 sees none of the three bots. The reason is mechanical. These crawlers do not render JavaScript, so they never fire your tag, and nothing comes back. GA4 measures humans with a browser; a crawler pulling raw HTML slips entirely under the radar.

That is also why Search Console’s “Generative AI performance” report, rolled out worldwide on August 31, 2026, is not enough: it gives impressions, not clicks, and says nothing about the crawl itself. On measuring the impact side, the topic is covered in detail in Google AI Overviews: Measuring the Impact on Organic Traffic. But to know who fetches what, there is only one source of truth: your server logs.

The pipeline: from logs to BigQuery

The good news is that the data already exists, it is just waiting to be used. Three possible sources depending on your stack: Cloudflare Logpush if you sit behind Cloudflare, your origin access logs (Cloud Run, Nginx, your CDN), or both cross-checked. You push those logs into BigQuery and you finally get a per-request view.

The minimal schema fits in five columns: timestamp, user_agent, client_ip, url_path, status_code. That is all the analysis below needs. On cost, we are talking a few cents to a few euros a month for a mid-sized site: BigQuery storage is billed on scanned volume, and these queries scan little if you partition by date. If you do not have a BigQuery dataset in place yet, the setup is described in Using the GA4 BigQuery Export; the log ingestion principle is the same, with a dedicated table.

The query: crawl volume by bot, by URL, week by week

Once the logs are in BigQuery, the first question is simple: who comes, how much, and on what. Here is the starting query, to adapt to your table name.

SELECT
  CASE
    WHEN user_agent LIKE '%GPTBot%'         THEN 'GPTBot (OpenAI train)'
    WHEN user_agent LIKE '%OAI-SearchBot%'  THEN 'OAI-SearchBot (search)'
    WHEN user_agent LIKE '%ChatGPT-User%'   THEN 'ChatGPT-User (live)'
    WHEN user_agent LIKE '%ClaudeBot%'      THEN 'ClaudeBot (Anthropic train)'
    WHEN user_agent LIKE '%Claude-SearchBot%' THEN 'Claude-SearchBot (search)'
    WHEN user_agent LIKE '%Claude-User%'    THEN 'Claude-User (live)'
    WHEN user_agent LIKE '%PerplexityBot%'  THEN 'PerplexityBot (search)'
    ELSE 'other'
  END AS bot,
  DATE_TRUNC(DATE(timestamp), WEEK) AS week,
  COUNT(*) AS hits,
  COUNT(DISTINCT url_path) AS distinct_urls
FROM `your_project.logs.access`
WHERE DATE(timestamp) >= DATE_SUB(CURRENT_DATE(), INTERVAL 8 WEEK)
  AND REGEXP_CONTAINS(user_agent, r'(?i)(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot)')
GROUP BY bot, week
ORDER BY week DESC, hits DESC;

You get, week by week, each bot’s volume and the number of distinct URLs it hits. Look at the hits / distinct_urls ratio: if it is very high, the bot keeps re-downloading pages that have not changed. Cloudflare documented that more than half of AI crawling goes precisely to re-downloading unchanged content, which is pure cost for you and zero added value. For more ready-to-use SQL recipes on your data, see The 10 Essential BigQuery Queries to Analyze GA4.

The step everyone skips: anti-spoofing

Here is the trap. Anyone can call themselves “GPTBot” in their user agent. If you decide on the user agent alone, you count impostors, and you risk blocking a legitimate bot while thinking you are hitting a fake one. Cloudflare in fact published, as early as August 2025, evidence of undeclared crawlers rotating their user agents and IPs to bypass robots.txt.

The fix: each vendor publishes its official IP ranges (OpenAI, Anthropic, Perplexity, Google). You load those ranges into a small BigQuery table and validate that the hit’s IP falls inside them before counting the bot as authentic. In practice, a join on the CIDR range, and a verified column set to true/false. Anything calling itself “GPTBot” from an IP outside the range goes into a “suspicious” bucket that you handle at the WAF level, not in your crawl policy. It is five minutes of work, and it changes your numbers completely.

The crawl-to-refer ratio: what each bot really costs

Now the heart of the matter, the part nobody covers. A bot that fetches a lot is not necessarily a problem, if it sends you traffic back. The right metric is the crawl-to-refer ratio: the number of pages the bot fetches divided by the number of visits it sends you.

You already have the numerator: your verified crawler hits, by vendor. The denominator is the sessions from GA4’s “AI Assistant” channel, split by source (ChatGPT, Perplexity, Claude). The method to build that channel cleanly and pull those referrals is detailed in Track ChatGPT, Gemini and Claude Traffic in GA4. You cross the two, vendor by vendor, and you get a telling ratio.

To give orders of magnitude, dated and to be taken as such: over the July 2026 window relayed from Cloudflare Radar, ClaudeBot accounted for around 18% of AI crawler requests versus roughly 10% for GPTBot, with crawl-to-refer ratios in the range of 2,200:1 on the Anthropic side and 200:1 on the OpenAI side. These numbers move enormously from one month and one source to the next, so do not treat them as stable truth: measure your own. The principle does not move, though: a ratio of several thousand to one is a bot that takes everything and gives nothing back.

The decision matrix: Allow, Charge, Block, bot by bot

You have the verified volumes and the ratios. You can finally decide, and the decision depends on your model. Here is the grid.

The underlying logic. In B2B, a citation in an AI answer brings highly qualified leads, so you keep the search and fetch bots wide open, and you tighten the training bots that give nothing back. In media, the content is the asset: the new Pay Per Use finally makes “Charge” credible, and the “block equals giving up the citation” trade-off plays out case by case (the topic is dug into in Measuring Your Brand’s Visibility in AI Answers). In e-commerce, be careful: blocking live fetch agents sometimes means blocking a buyer coming through an agent, an increasingly common case detailed in Agentic Commerce in GA4: Tracking ChatGPT and AI Mode Sales.

What to put in the monthly reporting

A crawl decision is not a one-shot, it is something you track. Add three metrics to your monthly reporting: verified crawl volume by bot, crawl-to-refer ratio by vendor, and the share of “wasted” crawling on unchanged pages. Wire it into your existing SEO rather than making it a silo, along the lines described in GA4 and Search Console: 5 SEO Analyses. Once the query is stable, industrialize it: monthly tracking goes on autopilot like any other report, see Automating Your Reporting with Looker Studio and BigQuery.

After September 15

This article carries a deadline, but the topic does not end on September 15. The Cloudflare switch is only a trigger: it forces you to decide now, on data you should have been looking at anyway. Once the deadline passes, the question becomes permanent, because new bots appear every quarter, IP ranges change, and ratios shift. Keep the logs-to-BigQuery pipeline in place, refresh your IP ranges, and re-read your matrix once a quarter.

The conclusion fits in one sentence: do not let a default setting decide for you. You have thirteen days, one query, and a matrix. Pull your logs, verify the IPs, compute your ratios, and judge bot by bot based on your model. It is faster to do than to read, and it is the only way to block the AI crawlers that cost you without sacrificing the ones that pay you back.