Scrape-to-Referral Ratio: 6,000 AI Scrapes for 1 Visit

Six thousand scrapes for a single visit sent back. That is the number Microsoft put front and center on August 13, 2026 when it launched the AI Scrape-to-Referral Ratio card in Clarity. It is spectacular, it is shareable, and it needs saying right away: it is not a published benchmark, it is a product screenshot. Six weeks earlier, on July 1, Cloudflare had opened its Attribution Business Insights with real per-operator ratios ranging from 118 to 1 up to nearly 50,000 to 1. When two players productize the same metric in six weeks, it means one thing: the scrape-to-referral ratio has just entered everyday vocabulary, and you will have to produce one for your own site.

The timing is no accident. September 15, 2026 was three days ago, and Cloudflare’s default blocking is now live. The question is no longer “should I get ready,” it is “what did this change on my site, and how do I measure it.” This is the direct sequel to the article on which crawlers to block before September 15, which stopped on the eve of the switch. This one is about the metric and its reliability, not about the mechanics of blocking. And it lands while 84% of the market is still ahead of the question: the IAB noted on August 4, 2026 that only 16% of brands track their AI visibility.

What the scrape-to-referral ratio actually measures

The scrape-to-referral ratio divides two numbers that do not come from the same place. The numerator is crawl requests: how many times an AI bot came to read your pages. You find them in your CDN or server logs. The denominator is referred sessions: how many human visitors reached you from an AI answer. You count those with a JavaScript tag, the analytics one.

Two different collection layers, then, and that is where the trouble is. Crawls are read server-side, at the moment of the HTTP request. Referrals are read browser-side, when the page loads. Nothing guarantees the two measurements cover the same set of domains, or the same time window. That is exactly why Clarity enforces its “mapped domains”: until the domain watched on the log side and the domain measured on the tag side are aligned, the ratio is arithmetic without meaning. A big number divided by another big number does not become a sound indicator just because someone gave it a name.

Keep this in mind for what follows: a ratio only means something if its numerator and denominator refer to the same site, over the same period. Almost nobody checks that, and that is where the surprises begin.

Getting the number: three paths

There are three ways to pull this ratio for your site, with very different prerequisites. Here is how they compare.

Clarity is free and has no traffic limit, which makes it very tempting. The blocker is not the price, it is the CDN: without Fastly, CloudFront, Cloudflare or a WordPress site with the up-to-date plugin, the card stays empty. A word of caution in passing, on the privacy side: Clarity sends your visitors’ browsing data to Microsoft. The product is free because the trade-off sits elsewhere. Recommending it without saying so would be the kind of shortcut this blog does not take.

Cloudflare gives a cleaner crawl count, since it reads traffic at the network level, but Attribution Business Insights is restricted to higher tiers and tracks referrals via UTM, so it misses anything that is not tagged.

That leaves the DIY method, logs plus BigQuery, the only one where you control the definition of both numerator and denominator. Collecting the logs and loading them into BigQuery is already covered in detail in the previous article, so there is no point rewriting it. What was missing is the join query that produces the ratio, operator by operator.

-- Scrape-to-referral ratio per operator, over 30 days
WITH crawls AS (
  SELECT operator, COUNT(*) AS n_crawls
  FROM `project.logs.crawl_events`
  WHERE event_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
  GROUP BY operator
),
referrals AS (
  SELECT operator, COUNT(DISTINCT session_id) AS n_referrals
  FROM `project.analytics.ai_sessions`
  WHERE event_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
  GROUP BY operator
)
SELECT
  c.operator,
  c.n_crawls,
  COALESCE(r.n_referrals, 0) AS n_referrals,
  SAFE_DIVIDE(c.n_crawls, r.n_referrals) AS scrape_to_referral
FROM crawls c
LEFT JOIN referrals r USING (operator)
ORDER BY scrape_to_referral DESC;

The join is on the operator, not on “the AIs” as a block. That is the only granularity that supports a decision, and also the only one that makes your ratio comparable to Cloudflare’s or Clarity’s.

Why your ratio is wrong (and in which direction)

Here is the passage neither Microsoft nor Cloudflare writes, because one sells blocking and the other sells its ecosystem. The ratio is wrong by construction, on both sides of the fraction. The good news is that it is wrong in a predictable direction.

An undercounted denominator

AI referrals are systematically undercounted, which makes the ratio look worse than it is. The causes are known and dated. Google confirmed on May 23, 2025 that AI Mode’s noreferrer attributes pushed clicks into Direct traffic. OpenAI only added UTM parameters to its links in June 2025. The Gemini app on iOS was still using an undocumented user-agent in October 2025. On the GA4 side, the AI Assistant channel only arrived on May 13, 2026, and the Source Group field in June 2026. The result: a meaningful share of AI-driven visits lands somewhere other than the “AI” bucket, and GA4’s native channel misses between 35% and 70% of them depending on setup. If this affects you, it is covered in depth in the guide to tracking ChatGPT, Gemini and Claude traffic in GA4.

As long as the denominator leaks, the imbalance is mechanically overstated. Your 6,000 to 1 might well be a 2,000 to 1 once the counting is fixed.

A misattributed numerator

On the other side, the numerator has its own flaws. A JavaScript tag never sees bots that do not run JavaScript, so if you try to count crawls tag-side, you miss most of them. And if you count them log-side, the opposite error waits: adding up all crawls without separating families means mixing things that do not owe you the same referral. A training crawler, an agent answering in real time and a search bot do not owe you the same visit. That three-family distinction is detailed in the blocking article; repeating it here would just be redoing it.

The operational conclusion is simple, and it is the most important point in this article: the ratio is a ranking indicator between operators, not an absolute value. The ranking is reliable, the magnitude is not. If Clarity tells you one operator scrapes ten times more than another for the same number of visits, trust the ranking. Do not trust the “6,000” to the digit.

What the ratio does not tell you: quality

A high ratio reads like a bad report card. Yet it ignores what matters most: what the rare visitors who do arrive actually do. The Clarity study of December 18, 2025, run across more than 1,200 publisher sites, gives a sharp contrast: 1.66% signup conversion for AI traffic versus 0.15% for classic search, with 155% growth over eight months for less than 1% of visitors. Few people, but people who act.

Be careful, though, not to turn that figure into a new benchmark. The variance between sectors is enormous, from “below parity” to “several times search” depending on the source. No sector average is usable as is. That is precisely the value of session recordings filtered by AI source: they replace a borrowed average with an observation of your own visitors. So the ratio should be read in three steps, in this order: volume first (how many real referrals), the ratio next (the operator’s rank), post-click behavior last (do these people convert). It is that reading, not the ratio alone, that makes the number the missing metric alongside the four already described in the guide to measuring your brand’s visibility in AI answers.

The decision grid

The ratio does not decide for you, it sorts. Here is how I arbitrate, operator by operator, once the number is in hand.

The last case is the most expensive trap. A mixed bot crawls for both search and training: blocking it on the basis of a bad ratio pushes you out of Google. The exact mechanics for telling them apart and blocking without collateral damage are in the previous article.

And keep a guardrail in mind before blocking anything: a study from Rutgers Business School and Wharton found that publishers who cut AI crawlers via robots.txt lost 23.1% of their monthly visits, with no proportionate protection in return. The ratio tells you who scrapes a lot; it does not tell you what blocking costs. Those are two separate questions.

What to do with it

The scrape-to-referral ratio is a tool for negotiation and prioritization, not a verdict. In an audit, it serves three purposes: ranking operators so you know where to start, spotting the ones that take without giving back, and putting a concrete number on the table when you discuss a content compensation deal. Nothing more, and that is already plenty.

The takeaway: compute it on your own data rather than copying a vendor’s benchmark, fix the denominator first, read the ranking and not the magnitude, then always cross it with post-click behavior before cutting anything. And if you sell online, remember that a referral is no longer the only possible counterpart to a crawl: part of the value now flows through agentic commerce, which is tracked separately in GA4. The spectacular number makes a good headline. The decision gets made operator by operator.