LogoAgentHunter
  • Submit
  • Industries
  • Categories
  • Agency
Logo
LogoAgentHunter

Discover, Compare, and Leverage the Best AI Agents

Featured On

Featured on yo.directory
yo.directory
Featured on yo.directory
Featured on Startup Fame
Startup Fame
Featured on Startup Fame
AIStage
Listed on AIStage
Sprunkid
Featured on Sprunkid
Featured on Twelve Tools
Twelve Tools
Featured on Twelve Tools
Listed on Turbo0
Turbo0
Listed on Turbo0
Featured on Product Hunt
Product Hunt
Featured on Product Hunt
Game Sprunki
Featured on Game Sprunki
AI Toolz Dir
Featured on AI Toolz Dir
Featured on Microlaunch
Microlaunch
Featured on Microlaunch
Featured on Fazier
Fazier
Featured on Fazier
Featured on Techbase Directory
Techbase Directory
Featured on Techbase Directory
backlinkdirs
Featured on Backlink Dirs
Featured on SideProjectors
SideProjectors
Featured on SideProjectors
Submit AI Tools
Featured on Submit AI Tools
AI Hunt
Featured on AI Hunt
Featured on Dang.ai
Dang.ai
Featured on Dang.ai
Featured on AI Finder
AI Finder
Featured on AI Finder
Featured on LaunchIgniter
LaunchIgniter
Featured on LaunchIgniter
Imglab
Featured on Imglab
AI138
Featured on AI138
600.tools
Featured on 600.tools
Featured Tool
Featured on Featured Tool
Dirs.cc
Featured on Dirs.cc
Ant Directory
Featured on Ant Directory
Featured on MagicBox.tools
MagicBox.tools
Featured on MagicBox.tools
Featured on Code.market
Code.market
Featured on Code.market
Featured on LaunchBoard
LaunchBoard
Featured on LaunchBoard
Genify
Featured on Genify
Copyright © 2025 All Rights Reserved.
Product
  • AI Agents Directory
  • AI Agent Glossary
  • Industries
  • Categories
Resources
  • AI Agentic Workflows
  • Blog
  • News
  • Submit
  • Coummunity
  • Ebooks
Company
  • About Us
  • Privacy Policy
  • Terms of Service
  • Sitemap
Friend Links
  • AI Music API
  • ImaginePro AI
  • Dog Names
  • Readdit Analytics
Back to News List

Cloudflare blocks AI crawlers by default to protect publishers

July 2, 2025•Thomas Claburn•Original Link•3 minutes
AI
Publishers
Cloudflare

Cloudflare is blocking AI web crawlers by default to help publishers get paid for their content as the old search deal breaks down.

Cloudflare has taken a bold step to block AI web crawlers by default, positioning itself as a gatekeeper to protect online publishers from what it calls the "stripmining" of web content by AI firms. The move comes as the traditional bargain between search engines and content creators—where traffic was exchanged for indexing—has broken down in the era of AI-driven search.

The Broken Deal

CEO Matthew Prince explained in a blog post that Google's original deal with publishers, where traffic was sent in exchange for content access, no longer holds. AI-driven search services, like those from Google, Microsoft, OpenAI, and Anthropic, now consume vast amounts of web content without sending proportional traffic back to publishers. This shift has left publishers struggling to monetize their work.

Prince argued, "The problem is whether you create content to sell ads, sell subscriptions, or just to know that people value what you've created, an AI-driven web doesn't reward content creators the way that the old search-driven web did."

The Data Behind the Decision

Cloudflare's data reveals stark disparities in how AI crawlers operate. For example, Anthropic’s Claude made 71,000 HTML page requests for every referral it sent back to websites. Other AI firms showed similar imbalances:

  • OpenAI: 1,600:1
  • Perplexity: 202.4:1
  • Microsoft: 40:1
  • Google: 9.4:1

Legacy search crawlers, in contrast, scanned content only a few times per visitor sent. The new AI-driven model, Cloudflare argues, undermines publisher revenue.

A New Payment Model

To address this, Cloudflare is launching Pay per crawl, a service in private beta that lets publishers charge AI crawlers for access. The system works like a tollbooth:

  • AI crawlers must present payment intent via request headers for access (HTTP 200).
  • Otherwise, they receive a 402 Payment Required response.

Publishers can choose to block crawlers, allow specific ones, or set pricing. Cloudflare acts as the "Merchant of Record" for these transactions.

Legal and Industry Context

The move comes as courts have largely sided with AI firms, ruling that training on publicly available data qualifies as fair use. Meanwhile, some publishers, like HarperCollins, have struck direct deals with AI companies—Microsoft paid $5,000 for a three-year license to use HarperCollins' content.

The Future of the Open Web

Cloudflare's initiative could make the web less open but may be necessary to sustain quality content creation. As Prince put it, "The web is being stripmined by AI crawlers." The question now is whether publishers will embrace this new gatekeeper role—and whether AI firms will pay up.

For more details, check Cloudflare's Pay per crawl signup page.

Related News

August 18, 2025•Kaydence Shum

Lenovo Wins Frost Sullivan 2025 Asia-Pacific AI Services Leadership Award

Lenovo earns Frost Sullivan's 2025 Asia-Pacific AI Services Customer Value Leadership Recognition for its value-driven innovation and real-world AI impact.

AI
Lenovo
Asia-Pacific
August 18, 2025•Unknown

Baidu Wenku GenFlow 2.0 Revolutionizes AI Agents with Multi-Agent Architecture

Baidu Wenku's GenFlow 2.0 introduces a multi-agent system for parallel task processing, integrating with Cangzhou OS to enhance efficiency and redefine AI workflows.

AI
MultiAgent
Baidu

About the Author

Michael Rodriguez

Michael Rodriguez

AI Technology Journalist

Veteran technology journalist with 12 years of focus on AI industry reporting. Former AI section editor at TechCrunch, now freelance writer contributing in-depth AI industry analysis to renowned media outlets like Wired and The Verge. Has keen insights into AI startups and emerging technology trends.

Expertise

AI Industry Analysis
Startup Ecosystem
Technology Trends
Product Reviews
Experience
12 years
Publications
800+
Credentials
2
LinkedInTwitter

Agent Newsletter

Get Agentic Newsletter Today

Subscribe to our newsletter for the latest news and updates