Cloudflare blocks AI crawlers by default to protect publishers

Cloudflare has taken a bold step to block AI web crawlers by default, positioning itself as a gatekeeper to protect online publishers from what it calls the "stripmining" of web content by AI firms. The move comes as the traditional bargain between search engines and content creators—where traffic was exchanged for indexing—has broken down in the era of AI-driven search.

The Broken Deal

CEO Matthew Prince explained in a blog post that Google's original deal with publishers, where traffic was sent in exchange for content access, no longer holds. AI-driven search services, like those from Google, Microsoft, OpenAI, and Anthropic, now consume vast amounts of web content without sending proportional traffic back to publishers. This shift has left publishers struggling to monetize their work.

Prince argued, "The problem is whether you create content to sell ads, sell subscriptions, or just to know that people value what you've created, an AI-driven web doesn't reward content creators the way that the old search-driven web did."

The Data Behind the Decision

Cloudflare's data reveals stark disparities in how AI crawlers operate. For example, Anthropic’s Claude made 71,000 HTML page requests for every referral it sent back to websites. Other AI firms showed similar imbalances:

• OpenAI: 1,600:1
• Perplexity: 202.4:1
• Microsoft: 40:1
• Google: 9.4:1

Legacy search crawlers, in contrast, scanned content only a few times per visitor sent. The new AI-driven model, Cloudflare argues, undermines publisher revenue.

A New Payment Model

To address this, Cloudflare is launching Pay per crawl, a service in private beta that lets publishers charge AI crawlers for access. The system works like a tollbooth:

• AI crawlers must present payment intent via request headers for access (HTTP 200).
• Otherwise, they receive a 402 Payment Required response.

Publishers can choose to block crawlers, allow specific ones, or set pricing. Cloudflare acts as the "Merchant of Record" for these transactions.

Legal and Industry Context

The move comes as courts have largely sided with AI firms, ruling that training on publicly available data qualifies as fair use. Meanwhile, some publishers, like HarperCollins, have struck direct deals with AI companies—Microsoft paid $5,000 for a three-year license to use HarperCollins' content.

The Future of the Open Web

Cloudflare's initiative could make the web less open but may be necessary to sustain quality content creation. As Prince put it, "The web is being stripmined by AI crawlers." The question now is whether publishers will embrace this new gatekeeper role—and whether AI firms will pay up.

For more details, check Cloudflare's Pay per crawl signup page.